Aug 05 16:21:24.727771 ip-10-0-132-37 systemd[1]: Starting Kubernetes Kubelet... Aug 05 16:21:25.118619 ip-10-0-132-37 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 05 16:21:25.118619 ip-10-0-132-37 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 05 16:21:25.118619 ip-10-0-132-37 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 05 16:21:25.118619 ip-10-0-132-37 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 05 16:21:25.118619 ip-10-0-132-37 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 05 16:21:25.119327 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.118678 2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 05 16:21:25.122759 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122742 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 16:21:25.122759 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122758 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 16:21:25.122759 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122761 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 16:21:25.122855 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122765 2573 feature_gate.go:328] unrecognized feature gate: Example Aug 05 16:21:25.122855 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122768 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 16:21:25.122855 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122771 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 16:21:25.122855 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122774 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 16:21:25.122855 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122777 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 16:21:25.122855 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122781 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 16:21:25.122855 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122784 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 16:21:25.122855 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122787 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 16:21:25.122855 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122790 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 16:21:25.122855 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122793 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 16:21:25.122855 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122796 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 16:21:25.122855 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122799 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 16:21:25.122855 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122802 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 16:21:25.122855 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122805 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 16:21:25.122855 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122808 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 16:21:25.122855 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122811 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 16:21:25.122855 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122813 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 16:21:25.122855 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122816 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 16:21:25.122855 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122824 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 16:21:25.122855 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122827 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 16:21:25.123334 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122829 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 16:21:25.123334 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122832 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 16:21:25.123334 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122834 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 16:21:25.123334 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122837 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 16:21:25.123334 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122840 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 16:21:25.123334 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122842 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 16:21:25.123334 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122844 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 16:21:25.123334 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122847 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 16:21:25.123334 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122850 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 16:21:25.123334 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122852 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 16:21:25.123334 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122855 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 16:21:25.123334 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122857 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 16:21:25.123334 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122860 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 16:21:25.123334 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122863 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 16:21:25.123334 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122866 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 16:21:25.123334 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122868 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 16:21:25.123334 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122871 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 16:21:25.123334 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122874 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 16:21:25.123334 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122877 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 16:21:25.123845 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122879 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 16:21:25.123845 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122884 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 16:21:25.123845 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122888 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 16:21:25.123845 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122892 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 16:21:25.123845 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122895 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 16:21:25.123845 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122897 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 16:21:25.123845 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122900 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 16:21:25.123845 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122903 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 16:21:25.123845 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122905 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 16:21:25.123845 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122909 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 16:21:25.123845 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122913 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 16:21:25.123845 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122915 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 16:21:25.123845 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122918 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 16:21:25.123845 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122921 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 16:21:25.123845 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122925 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 16:21:25.123845 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122927 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 16:21:25.123845 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122930 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 16:21:25.123845 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122934 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 16:21:25.123845 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122936 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 16:21:25.124302 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122939 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 16:21:25.124302 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122942 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 16:21:25.124302 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122945 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 16:21:25.124302 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122948 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 16:21:25.124302 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122951 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 16:21:25.124302 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122953 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 16:21:25.124302 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122956 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 16:21:25.124302 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122958 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 16:21:25.124302 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122961 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 16:21:25.124302 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122963 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 16:21:25.124302 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122966 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 16:21:25.124302 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122968 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 16:21:25.124302 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122972 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 16:21:25.124302 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122974 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 16:21:25.124302 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122977 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 16:21:25.124302 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122979 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 16:21:25.124302 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122982 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 16:21:25.124302 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122984 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 16:21:25.124302 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122987 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 16:21:25.124302 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122990 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 16:21:25.124797 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122992 2573 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 16:21:25.124797 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122996 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 16:21:25.124797 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.122999 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 16:21:25.124797 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123002 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 16:21:25.124797 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123005 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 16:21:25.124797 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123008 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 16:21:25.124797 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123011 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 16:21:25.124797 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123014 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 16:21:25.124797 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123023 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 16:21:25.124797 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123026 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 16:21:25.124797 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123029 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 16:21:25.124797 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123031 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 16:21:25.124797 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123034 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 16:21:25.124797 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123037 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 16:21:25.124797 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123039 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 16:21:25.124797 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123042 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 16:21:25.124797 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123044 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 16:21:25.124797 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123047 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 16:21:25.124797 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123815 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 16:21:25.125251 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123822 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 16:21:25.125251 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123825 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 16:21:25.125251 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123828 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 16:21:25.125251 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123831 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 16:21:25.125251 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123834 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 16:21:25.125251 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123836 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 16:21:25.125251 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123839 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 16:21:25.125251 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123841 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 16:21:25.125251 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123844 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 16:21:25.125251 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123847 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 16:21:25.125251 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123850 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 16:21:25.125251 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123853 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 16:21:25.125251 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123856 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 16:21:25.125251 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123859 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 16:21:25.125251 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123862 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 16:21:25.125251 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123864 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 16:21:25.125251 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123867 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 16:21:25.125251 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123869 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 16:21:25.125251 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123872 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 16:21:25.125251 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123874 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 16:21:25.125753 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123878 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 16:21:25.125753 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123881 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 16:21:25.125753 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123883 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 16:21:25.125753 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123887 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 16:21:25.125753 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123891 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 16:21:25.125753 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123894 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 16:21:25.125753 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123897 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 16:21:25.125753 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123900 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 16:21:25.125753 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123902 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 16:21:25.125753 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123905 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 16:21:25.125753 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123907 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 16:21:25.125753 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123910 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 16:21:25.125753 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123912 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 16:21:25.125753 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123915 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 16:21:25.125753 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123917 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 16:21:25.125753 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123919 2573 feature_gate.go:328] unrecognized feature gate: Example Aug 05 16:21:25.125753 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123922 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 16:21:25.125753 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123924 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 16:21:25.125753 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123927 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 16:21:25.125753 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123929 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 16:21:25.126284 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123932 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 16:21:25.126284 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123935 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 16:21:25.126284 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123937 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 16:21:25.126284 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123940 2573 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 16:21:25.126284 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123943 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 16:21:25.126284 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123945 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 16:21:25.126284 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123948 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 16:21:25.126284 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123950 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 16:21:25.126284 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123953 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 16:21:25.126284 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123955 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 16:21:25.126284 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123958 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 16:21:25.126284 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123960 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 16:21:25.126284 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123963 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 16:21:25.126284 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123966 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 16:21:25.126284 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123969 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 16:21:25.126284 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123972 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 16:21:25.126284 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123974 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 16:21:25.126284 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123977 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 16:21:25.126284 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123979 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 16:21:25.126284 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123982 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 16:21:25.126916 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123984 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 16:21:25.126916 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123987 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 16:21:25.126916 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123989 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 16:21:25.126916 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123991 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 16:21:25.126916 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123994 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 16:21:25.126916 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123996 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 16:21:25.126916 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.123999 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 16:21:25.126916 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124001 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 16:21:25.126916 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124005 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 16:21:25.126916 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124007 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 16:21:25.126916 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124010 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 16:21:25.126916 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124013 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 16:21:25.126916 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124016 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 16:21:25.126916 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124020 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 16:21:25.126916 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124024 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 16:21:25.126916 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124027 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 16:21:25.126916 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124030 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 16:21:25.126916 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124036 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 16:21:25.126916 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124039 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 16:21:25.127406 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124042 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 16:21:25.127406 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124045 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 16:21:25.127406 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124048 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 16:21:25.127406 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124050 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 16:21:25.127406 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124053 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 16:21:25.127406 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124056 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 16:21:25.127406 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124059 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 16:21:25.127406 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124062 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 16:21:25.127406 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124064 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 16:21:25.127406 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124067 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 16:21:25.127406 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124070 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 16:21:25.127406 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124072 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 16:21:25.127406 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124075 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 16:21:25.127406 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124077 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 16:21:25.127406 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124080 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 16:21:25.127406 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124082 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 16:21:25.127406 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124085 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 16:21:25.127406 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124087 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 16:21:25.127406 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124090 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 16:21:25.127406 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124478 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 16:21:25.127892 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124484 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 16:21:25.127892 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124488 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 16:21:25.127892 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124491 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 16:21:25.127892 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124494 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 16:21:25.127892 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124497 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 16:21:25.127892 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124499 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 16:21:25.127892 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124502 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 16:21:25.127892 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124505 2573 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 16:21:25.127892 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124508 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 16:21:25.127892 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124511 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 16:21:25.127892 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124513 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 16:21:25.127892 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124516 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 16:21:25.127892 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124518 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 16:21:25.127892 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124521 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 16:21:25.127892 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124524 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 16:21:25.127892 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124526 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 16:21:25.127892 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124529 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 16:21:25.127892 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124533 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 16:21:25.127892 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124537 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 16:21:25.127892 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124540 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 16:21:25.128400 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124543 2573 feature_gate.go:328] unrecognized feature gate: Example Aug 05 16:21:25.128400 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124546 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 16:21:25.128400 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124550 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 16:21:25.128400 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124553 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 16:21:25.128400 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124557 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 16:21:25.128400 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124560 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 16:21:25.128400 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124562 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 16:21:25.128400 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124565 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 16:21:25.128400 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124568 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 16:21:25.128400 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124571 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 16:21:25.128400 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124574 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 16:21:25.128400 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124576 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 16:21:25.128400 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124578 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 16:21:25.128400 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124581 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 16:21:25.128400 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124585 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 16:21:25.128400 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124587 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 16:21:25.128400 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124590 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 16:21:25.128400 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124592 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 16:21:25.128400 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124595 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 16:21:25.128870 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124597 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 16:21:25.128870 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124600 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 16:21:25.128870 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124603 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 16:21:25.128870 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124605 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 16:21:25.128870 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124609 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 16:21:25.128870 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124612 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 16:21:25.128870 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124615 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 16:21:25.128870 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124617 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 16:21:25.128870 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124620 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 16:21:25.128870 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124623 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 16:21:25.128870 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124626 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 16:21:25.128870 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124629 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 16:21:25.128870 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124632 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 16:21:25.128870 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124635 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 16:21:25.128870 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124637 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 16:21:25.128870 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124640 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 16:21:25.128870 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124643 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 16:21:25.128870 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124646 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 16:21:25.128870 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124648 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 16:21:25.129331 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124651 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 16:21:25.129331 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124653 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 16:21:25.129331 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124656 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 16:21:25.129331 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124658 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 16:21:25.129331 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124665 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 16:21:25.129331 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124668 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 16:21:25.129331 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124670 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 16:21:25.129331 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124673 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 16:21:25.129331 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124676 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 16:21:25.129331 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124679 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 16:21:25.129331 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124682 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 16:21:25.129331 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124684 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 16:21:25.129331 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124687 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 16:21:25.129331 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124690 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 16:21:25.129331 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124693 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 16:21:25.129331 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124696 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 16:21:25.129331 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124699 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 16:21:25.129331 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124701 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 16:21:25.129331 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124703 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 16:21:25.129331 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124706 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 16:21:25.129821 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124708 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 16:21:25.129821 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124711 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 16:21:25.129821 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124713 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 16:21:25.129821 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124716 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 16:21:25.129821 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124718 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 16:21:25.129821 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124720 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 16:21:25.129821 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124723 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 16:21:25.129821 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124725 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 16:21:25.129821 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124728 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 16:21:25.129821 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124733 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 16:21:25.129821 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124736 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 16:21:25.129821 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124738 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 16:21:25.129821 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124741 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 16:21:25.129821 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124743 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 16:21:25.129821 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124745 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 16:21:25.129821 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124748 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 16:21:25.129821 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124751 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 16:21:25.129821 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124754 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 16:21:25.129821 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124756 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 16:21:25.129821 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.124759 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 16:21:25.130308 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124825 2573 flags.go:64] FLAG: --address="0.0.0.0" Aug 05 16:21:25.130308 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124832 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 05 16:21:25.130308 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124837 2573 flags.go:64] FLAG: --anonymous-auth="true" Aug 05 16:21:25.130308 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124842 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 05 16:21:25.130308 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124846 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 05 16:21:25.130308 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124849 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 05 16:21:25.130308 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124853 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 05 16:21:25.130308 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124857 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 05 16:21:25.130308 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124861 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 05 16:21:25.130308 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124864 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 05 16:21:25.130308 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124868 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 05 16:21:25.130308 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124871 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 05 16:21:25.130308 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124874 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 05 16:21:25.130308 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124877 2573 flags.go:64] FLAG: --cgroup-root="" Aug 05 16:21:25.130308 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124880 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 05 16:21:25.130308 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124883 2573 flags.go:64] FLAG: --client-ca-file="" Aug 05 16:21:25.130308 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124886 2573 flags.go:64] FLAG: --cloud-provider="external" Aug 05 16:21:25.130308 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124889 2573 flags.go:64] FLAG: --cluster-dns="[]" Aug 05 16:21:25.130308 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124896 2573 flags.go:64] FLAG: --cluster-domain="" Aug 05 16:21:25.130308 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124898 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 05 16:21:25.130308 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124902 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 05 16:21:25.130308 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124907 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 05 16:21:25.130308 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124911 2573 flags.go:64] FLAG: --container-log-max-files="5" Aug 05 16:21:25.130308 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124914 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 05 16:21:25.130894 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124917 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 05 16:21:25.130894 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124920 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 05 16:21:25.130894 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124923 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 05 16:21:25.130894 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124926 2573 flags.go:64] FLAG: --contention-profiling="false" Aug 05 16:21:25.130894 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124931 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 05 16:21:25.130894 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124934 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 05 16:21:25.130894 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124937 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 05 16:21:25.130894 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124940 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 05 16:21:25.130894 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124947 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 05 16:21:25.130894 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124950 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 05 16:21:25.130894 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124953 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 05 16:21:25.130894 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124956 2573 flags.go:64] FLAG: --enable-load-reader="false" Aug 05 16:21:25.130894 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124958 2573 flags.go:64] FLAG: --enable-server="true" Aug 05 16:21:25.130894 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124961 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 05 16:21:25.130894 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124968 2573 flags.go:64] FLAG: --event-burst="100" Aug 05 16:21:25.130894 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124971 2573 flags.go:64] FLAG: --event-qps="50" Aug 05 16:21:25.130894 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124974 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 05 16:21:25.130894 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124982 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 05 16:21:25.130894 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124986 2573 flags.go:64] FLAG: --eviction-hard="" Aug 05 16:21:25.130894 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124989 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 05 16:21:25.130894 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124992 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 05 16:21:25.130894 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124995 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 05 16:21:25.130894 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.124998 2573 flags.go:64] FLAG: --eviction-soft="" Aug 05 16:21:25.130894 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125001 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 05 16:21:25.130894 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125004 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 05 16:21:25.131772 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125007 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 05 16:21:25.131772 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125010 2573 flags.go:64] FLAG: --experimental-mounter-path="" Aug 05 16:21:25.131772 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125012 2573 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 05 16:21:25.131772 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125015 2573 flags.go:64] FLAG: --fail-swap-on="true" Aug 05 16:21:25.131772 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125019 2573 flags.go:64] FLAG: --feature-gates="" Aug 05 16:21:25.131772 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125023 2573 flags.go:64] FLAG: --file-check-frequency="20s" Aug 05 16:21:25.131772 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125026 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 05 16:21:25.131772 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125029 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 05 16:21:25.131772 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125032 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 05 16:21:25.131772 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125035 2573 flags.go:64] FLAG: --healthz-port="10248" Aug 05 16:21:25.131772 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125038 2573 flags.go:64] FLAG: --help="false" Aug 05 16:21:25.131772 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125042 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-132-37.ec2.internal" Aug 05 16:21:25.131772 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125045 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 05 16:21:25.131772 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125048 2573 flags.go:64] FLAG: --http-check-frequency="20s" Aug 05 16:21:25.131772 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125051 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 05 16:21:25.131772 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125054 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 05 16:21:25.131772 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125057 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 05 16:21:25.131772 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125060 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 05 16:21:25.131772 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125063 2573 flags.go:64] FLAG: --image-service-endpoint="" Aug 05 16:21:25.131772 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125065 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 05 16:21:25.131772 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125068 2573 flags.go:64] FLAG: --kube-api-burst="100" Aug 05 16:21:25.131772 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125071 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 05 16:21:25.131772 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125075 2573 flags.go:64] FLAG: --kube-api-qps="50" Aug 05 16:21:25.131772 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125078 2573 flags.go:64] FLAG: --kube-reserved="" Aug 05 16:21:25.132803 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125080 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 05 16:21:25.132803 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125088 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 05 16:21:25.132803 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125092 2573 flags.go:64] FLAG: --kubelet-cgroups="" Aug 05 16:21:25.132803 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125094 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 05 16:21:25.132803 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125097 2573 flags.go:64] FLAG: --lock-file="" Aug 05 16:21:25.132803 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125100 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 05 16:21:25.132803 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125102 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 05 16:21:25.132803 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125105 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 05 16:21:25.132803 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125111 2573 flags.go:64] FLAG: --log-json-split-stream="false" Aug 05 16:21:25.132803 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125114 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 05 16:21:25.132803 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125117 2573 flags.go:64] FLAG: --log-text-split-stream="false" Aug 05 16:21:25.132803 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125120 2573 flags.go:64] FLAG: --logging-format="text" Aug 05 16:21:25.132803 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125123 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 05 16:21:25.132803 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125127 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 05 16:21:25.132803 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125129 2573 flags.go:64] FLAG: --manifest-url="" Aug 05 16:21:25.132803 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125132 2573 flags.go:64] FLAG: --manifest-url-header="" Aug 05 16:21:25.132803 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125136 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 05 16:21:25.132803 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125139 2573 flags.go:64] FLAG: --max-open-files="1000000" Aug 05 16:21:25.132803 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125143 2573 flags.go:64] FLAG: --max-pods="110" Aug 05 16:21:25.132803 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125151 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 05 16:21:25.132803 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125154 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 05 16:21:25.132803 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125157 2573 flags.go:64] FLAG: --memory-manager-policy="None" Aug 05 16:21:25.132803 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125160 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 05 16:21:25.132803 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125163 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 05 16:21:25.132803 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125165 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 05 16:21:25.133767 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125169 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 05 16:21:25.133767 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125176 2573 flags.go:64] FLAG: --node-status-max-images="50" Aug 05 16:21:25.133767 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125178 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 05 16:21:25.133767 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125181 2573 flags.go:64] FLAG: --oom-score-adj="-999" Aug 05 16:21:25.133767 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125184 2573 flags.go:64] FLAG: --pod-cidr="" Aug 05 16:21:25.133767 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125187 2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 05 16:21:25.133767 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125192 2573 flags.go:64] FLAG: --pod-manifest-path="" Aug 05 16:21:25.133767 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125195 2573 flags.go:64] FLAG: --pod-max-pids="-1" Aug 05 16:21:25.133767 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125198 2573 flags.go:64] FLAG: --pods-per-core="0" Aug 05 16:21:25.133767 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125208 2573 flags.go:64] FLAG: --port="10250" Aug 05 16:21:25.133767 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125211 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 05 16:21:25.133767 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125214 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-080973fa84ad8c043" Aug 05 16:21:25.133767 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125217 2573 flags.go:64] FLAG: --qos-reserved="" Aug 05 16:21:25.133767 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125220 2573 flags.go:64] FLAG: --read-only-port="10255" Aug 05 16:21:25.133767 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125223 2573 flags.go:64] FLAG: --register-node="true" Aug 05 16:21:25.133767 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125226 2573 flags.go:64] FLAG: --register-with-taints="" Aug 05 16:21:25.133767 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125230 2573 flags.go:64] FLAG: --registry-burst="10" Aug 05 16:21:25.133767 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125233 2573 flags.go:64] FLAG: --registry-qps="5" Aug 05 16:21:25.133767 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125235 2573 flags.go:64] FLAG: --reserved-cpus="" Aug 05 16:21:25.133767 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125239 2573 flags.go:64] FLAG: --reserved-memory="" Aug 05 16:21:25.133767 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125243 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 05 16:21:25.133767 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125246 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 05 16:21:25.133767 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125249 2573 flags.go:64] FLAG: --rotate-certificates="false" Aug 05 16:21:25.133767 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125252 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 05 16:21:25.133767 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125255 2573 flags.go:64] FLAG: --runonce="false" Aug 05 16:21:25.134484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125258 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 05 16:21:25.134484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125261 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 05 16:21:25.134484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125265 2573 flags.go:64] FLAG: --seccomp-default="false" Aug 05 16:21:25.134484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125268 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 05 16:21:25.134484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125271 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 05 16:21:25.134484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125274 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 05 16:21:25.134484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125277 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 05 16:21:25.134484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125280 2573 flags.go:64] FLAG: --storage-driver-password="root" Aug 05 16:21:25.134484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125282 2573 flags.go:64] FLAG: --storage-driver-secure="false" Aug 05 16:21:25.134484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125286 2573 flags.go:64] FLAG: --storage-driver-table="stats" Aug 05 16:21:25.134484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125288 2573 flags.go:64] FLAG: --storage-driver-user="root" Aug 05 16:21:25.134484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125291 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 05 16:21:25.134484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125294 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 05 16:21:25.134484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125298 2573 flags.go:64] FLAG: --system-cgroups="" Aug 05 16:21:25.134484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125301 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 05 16:21:25.134484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125306 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 05 16:21:25.134484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125309 2573 flags.go:64] FLAG: --tls-cert-file="" Aug 05 16:21:25.134484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125312 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 05 16:21:25.134484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125316 2573 flags.go:64] FLAG: --tls-min-version="" Aug 05 16:21:25.134484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125319 2573 flags.go:64] FLAG: --tls-private-key-file="" Aug 05 16:21:25.134484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125322 2573 flags.go:64] FLAG: --topology-manager-policy="none" Aug 05 16:21:25.134484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125325 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 05 16:21:25.134484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125327 2573 flags.go:64] FLAG: --topology-manager-scope="container" Aug 05 16:21:25.134484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125330 2573 flags.go:64] FLAG: --v="2" Aug 05 16:21:25.134484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125334 2573 flags.go:64] FLAG: --version="false" Aug 05 16:21:25.135588 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125338 2573 flags.go:64] FLAG: --vmodule="" Aug 05 16:21:25.135588 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125342 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 05 16:21:25.135588 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.125345 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 05 16:21:25.135588 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125484 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 16:21:25.135588 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125488 2573 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 16:21:25.135588 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125491 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 16:21:25.135588 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125494 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 16:21:25.135588 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125497 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 16:21:25.135588 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125499 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 16:21:25.135588 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125503 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 16:21:25.135588 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125505 2573 feature_gate.go:328] unrecognized feature gate: Example Aug 05 16:21:25.135588 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125508 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 16:21:25.135588 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125510 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 16:21:25.135588 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125514 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 16:21:25.135588 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125518 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 16:21:25.135588 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125521 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 16:21:25.135588 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125524 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 16:21:25.135588 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125526 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 16:21:25.135588 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125529 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 16:21:25.135588 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125532 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 16:21:25.136204 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125534 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 16:21:25.136204 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125537 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 16:21:25.136204 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125539 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 16:21:25.136204 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125542 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 16:21:25.136204 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125545 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 16:21:25.136204 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125547 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 16:21:25.136204 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125550 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 16:21:25.136204 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125553 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 16:21:25.136204 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125555 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 16:21:25.136204 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125558 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 16:21:25.136204 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125560 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 16:21:25.136204 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125563 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 16:21:25.136204 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125565 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 16:21:25.136204 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125568 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 16:21:25.136204 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125570 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 16:21:25.136204 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125573 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 16:21:25.136204 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125575 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 16:21:25.136204 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125578 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 16:21:25.136204 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125580 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 16:21:25.136204 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125583 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 16:21:25.137041 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125585 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 16:21:25.137041 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125589 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 16:21:25.137041 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125593 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 16:21:25.137041 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125596 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 16:21:25.137041 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125599 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 16:21:25.137041 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125601 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 16:21:25.137041 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125604 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 16:21:25.137041 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125607 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 16:21:25.137041 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125610 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 16:21:25.137041 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125612 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 16:21:25.137041 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125615 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 16:21:25.137041 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125617 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 16:21:25.137041 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125620 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 16:21:25.137041 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125622 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 16:21:25.137041 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125625 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 16:21:25.137041 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125627 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 16:21:25.137041 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125630 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 16:21:25.137041 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125633 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 16:21:25.137041 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125636 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 16:21:25.137860 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125638 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 16:21:25.137860 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125641 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 16:21:25.137860 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125643 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 16:21:25.137860 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125646 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 16:21:25.137860 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125648 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 16:21:25.137860 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125651 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 16:21:25.137860 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125653 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 16:21:25.137860 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125656 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 16:21:25.137860 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125658 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 16:21:25.137860 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125661 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 16:21:25.137860 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125663 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 16:21:25.137860 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125666 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 16:21:25.137860 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125669 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 16:21:25.137860 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125671 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 16:21:25.137860 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125674 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 16:21:25.137860 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125676 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 16:21:25.137860 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125679 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 16:21:25.137860 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125681 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 16:21:25.137860 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125684 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 16:21:25.138576 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125686 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 16:21:25.138576 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125688 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 16:21:25.138576 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125691 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 16:21:25.138576 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125694 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 16:21:25.138576 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125696 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 16:21:25.138576 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125699 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 16:21:25.138576 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125701 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 16:21:25.138576 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125704 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 16:21:25.138576 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125707 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 16:21:25.138576 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125709 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 16:21:25.138576 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125712 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 16:21:25.138576 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125714 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 16:21:25.138576 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125718 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 16:21:25.138576 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125722 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 16:21:25.138576 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125725 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 16:21:25.138576 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125727 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 16:21:25.138576 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125730 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 16:21:25.138576 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125733 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 16:21:25.138576 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125736 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 16:21:25.139078 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125738 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 16:21:25.139078 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125741 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 16:21:25.139078 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125744 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 16:21:25.139078 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125746 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 16:21:25.139078 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.125749 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 16:21:25.139078 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.126256 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 05 16:21:25.139078 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.133577 2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 05 16:21:25.139078 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.133596 2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 05 16:21:25.139078 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.134316 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 05 16:21:25.139078 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.134330 2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 05 16:21:25.139078 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134433 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 16:21:25.139078 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134444 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 16:21:25.139078 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134452 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 16:21:25.139078 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134457 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 16:21:25.139460 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134462 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 16:21:25.139460 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134468 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 16:21:25.139460 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134472 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 16:21:25.139460 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134477 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 16:21:25.139460 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134481 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 16:21:25.139460 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134485 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 16:21:25.139460 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134490 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 16:21:25.139460 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134494 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 16:21:25.139460 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134498 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 16:21:25.139460 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134503 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 16:21:25.139460 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134507 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 16:21:25.139460 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134513 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 16:21:25.139460 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134519 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 16:21:25.139460 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134525 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 16:21:25.139460 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134530 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 16:21:25.139460 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134535 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 16:21:25.139460 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134540 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 16:21:25.139460 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134545 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 16:21:25.139460 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134550 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 16:21:25.139460 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134555 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 16:21:25.139996 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134559 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 16:21:25.139996 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134564 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 16:21:25.139996 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134568 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 16:21:25.139996 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134573 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 16:21:25.139996 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134577 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 16:21:25.139996 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134584 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 16:21:25.139996 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134588 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 16:21:25.139996 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134593 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 16:21:25.139996 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134597 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 16:21:25.139996 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134601 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 16:21:25.139996 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134605 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 16:21:25.139996 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134609 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 16:21:25.139996 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134614 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 16:21:25.139996 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134618 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 16:21:25.139996 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134622 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 16:21:25.139996 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134626 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 16:21:25.139996 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134630 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 16:21:25.139996 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134634 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 16:21:25.139996 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134639 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 16:21:25.139996 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134643 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 16:21:25.140499 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134647 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 16:21:25.140499 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134651 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 16:21:25.140499 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134656 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 16:21:25.140499 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134661 2573 feature_gate.go:328] unrecognized feature gate: Example Aug 05 16:21:25.140499 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134665 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 16:21:25.140499 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134669 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 16:21:25.140499 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134673 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 16:21:25.140499 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134677 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 16:21:25.140499 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134682 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 16:21:25.140499 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134686 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 16:21:25.140499 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134690 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 16:21:25.140499 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134694 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 16:21:25.140499 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134698 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 16:21:25.140499 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134702 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 16:21:25.140499 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134707 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 16:21:25.140499 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134711 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 16:21:25.140499 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134716 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 16:21:25.140499 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134720 2573 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 16:21:25.140499 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134725 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 16:21:25.140499 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134730 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 16:21:25.140499 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134735 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 16:21:25.141066 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134739 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 16:21:25.141066 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134743 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 16:21:25.141066 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134747 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 16:21:25.141066 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134751 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 16:21:25.141066 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134756 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 16:21:25.141066 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134760 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 16:21:25.141066 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134764 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 16:21:25.141066 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134768 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 16:21:25.141066 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134775 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 16:21:25.141066 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134780 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 16:21:25.141066 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134784 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 16:21:25.141066 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134788 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 16:21:25.141066 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134792 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 16:21:25.141066 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134797 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 16:21:25.141066 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134801 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 16:21:25.141066 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134805 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 16:21:25.141066 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134809 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 16:21:25.141066 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134814 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 16:21:25.141066 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134818 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 16:21:25.141624 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134822 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 16:21:25.141624 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134826 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 16:21:25.141624 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134830 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 16:21:25.141624 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134834 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 16:21:25.141624 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134838 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 16:21:25.141624 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134842 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 16:21:25.141624 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134846 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 16:21:25.141624 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134850 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 16:21:25.141624 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134854 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 16:21:25.141624 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134859 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 16:21:25.141624 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134863 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 16:21:25.141624 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134867 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 16:21:25.141624 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134872 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 16:21:25.141624 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134876 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 16:21:25.141624 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.134880 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 16:21:25.141985 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.134888 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 05 16:21:25.141985 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135080 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 16:21:25.141985 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135089 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 16:21:25.141985 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135094 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 16:21:25.141985 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135099 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 16:21:25.141985 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135103 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 16:21:25.141985 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135107 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 16:21:25.141985 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135112 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 16:21:25.141985 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135116 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 16:21:25.141985 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135120 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 16:21:25.141985 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135125 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 16:21:25.141985 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135129 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 16:21:25.141985 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135133 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 16:21:25.141985 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135138 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 16:21:25.141985 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135142 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 16:21:25.142367 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135146 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 16:21:25.142367 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135150 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 16:21:25.142367 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135154 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 16:21:25.142367 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135158 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 16:21:25.142367 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135163 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 16:21:25.142367 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135167 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 16:21:25.142367 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135171 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 16:21:25.142367 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135178 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 16:21:25.142367 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135184 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 16:21:25.142367 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135188 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 16:21:25.142367 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135193 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 16:21:25.142367 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135197 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 16:21:25.142367 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135201 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 16:21:25.142367 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135205 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 16:21:25.142367 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135211 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 16:21:25.142367 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135215 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 16:21:25.142367 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135219 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 16:21:25.142367 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135223 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 16:21:25.142367 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135227 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 16:21:25.142867 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135231 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 16:21:25.142867 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135237 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 16:21:25.142867 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135243 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 16:21:25.142867 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135248 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 16:21:25.142867 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135252 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 16:21:25.142867 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135256 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 16:21:25.142867 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135261 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 16:21:25.142867 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135265 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 16:21:25.142867 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135269 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 16:21:25.142867 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135273 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 16:21:25.142867 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135277 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 16:21:25.142867 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135281 2573 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 16:21:25.142867 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135288 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 16:21:25.142867 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135294 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 16:21:25.142867 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135299 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 16:21:25.142867 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135303 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 16:21:25.142867 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135307 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 16:21:25.142867 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135312 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 16:21:25.142867 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135316 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 16:21:25.143330 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135320 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 16:21:25.143330 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135324 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 16:21:25.143330 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135328 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 16:21:25.143330 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135332 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 16:21:25.143330 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135336 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 16:21:25.143330 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135341 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 16:21:25.143330 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135345 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 16:21:25.143330 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135348 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 16:21:25.143330 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135352 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 16:21:25.143330 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135359 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 16:21:25.143330 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135363 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 16:21:25.143330 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135367 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 16:21:25.143330 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135386 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 16:21:25.143330 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135391 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 16:21:25.143330 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135396 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 16:21:25.143330 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135400 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 16:21:25.143330 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135404 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 16:21:25.143330 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135408 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 16:21:25.143330 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135412 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 16:21:25.143330 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135416 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 16:21:25.144004 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135420 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 16:21:25.144004 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135424 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 16:21:25.144004 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135428 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 16:21:25.144004 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135432 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 16:21:25.144004 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135436 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 16:21:25.144004 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135440 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 16:21:25.144004 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135445 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 16:21:25.144004 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135450 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 16:21:25.144004 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135454 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 16:21:25.144004 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135458 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 16:21:25.144004 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135462 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 16:21:25.144004 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135466 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 16:21:25.144004 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135470 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 16:21:25.144004 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135473 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 16:21:25.144004 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135478 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 16:21:25.144004 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135482 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 16:21:25.144004 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135489 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 16:21:25.144004 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135493 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 16:21:25.144004 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135497 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 16:21:25.144004 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135501 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 16:21:25.144813 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135505 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 16:21:25.144813 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135509 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 16:21:25.144813 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135515 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 16:21:25.144813 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135519 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 16:21:25.144813 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135523 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 16:21:25.144813 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135528 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 16:21:25.144813 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:25.135532 2573 feature_gate.go:328] unrecognized feature gate: Example Aug 05 16:21:25.144813 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.135540 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 05 16:21:25.144813 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.135704 2573 server.go:962] "Client rotation is on, will bootstrap in background" Aug 05 16:21:25.144813 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.138270 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 05 16:21:25.144813 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.139653 2573 server.go:1019] "Starting client certificate rotation" Aug 05 16:21:25.144813 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.139747 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 05 16:21:25.144813 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.141062 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 05 16:21:25.145298 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.144292 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 05 16:21:25.147271 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.147251 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 05 16:21:25.158311 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.158291 2573 log.go:25] "Validated CRI v1 runtime API" Aug 05 16:21:25.160310 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.160295 2573 log.go:25] "Validated CRI v1 image API" Aug 05 16:21:25.162192 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.162169 2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 05 16:21:25.164663 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.164642 2573 fs.go:135] Filesystem UUIDs: map[24ce5173-817e-4c52-b22a-a09d0ec02037:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ee188d13-00b1-450f-a102-fa5a843f8d5d:/dev/nvme0n1p4] Aug 05 16:21:25.164737 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.164662 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 05 16:21:25.170016 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.169999 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 05 16:21:25.170693 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.170587 2573 manager.go:216] Machine: {Timestamp:2026-08-05 16:21:25.168589216 +0000 UTC m=+0.325435108 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099947 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27f37a8d66df01fdc6150d9b7a8f7c SystemUUID:ec27f37a-8d66-df01-fdc6-150d9b7a8f7c BootID:a258cf74-2201-4c52-babf-75ca7be541a5 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:45:d1:44:29:35 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:45:d1:44:29:35 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:bf:e8:42:d6:74 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 05 16:21:25.170693 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.170689 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 05 16:21:25.170785 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.170770 2573 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 05 16:21:25.173628 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.173602 2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 05 16:21:25.173761 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.173630 2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-37.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 05 16:21:25.174381 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.174361 2573 topology_manager.go:138] "Creating topology manager with none policy" Aug 05 16:21:25.174421 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.174386 2573 container_manager_linux.go:309] "Creating device plugin manager" Aug 05 16:21:25.174421 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.174396 2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 05 16:21:25.175022 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.175005 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 05 16:21:25.175949 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.175939 2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 05 16:21:25.179517 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.179505 2573 state_mem.go:36] "Initialized new in-memory state store" Aug 05 16:21:25.179641 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.179630 2573 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 05 16:21:25.179681 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.179668 2573 kubelet.go:486] "Attempting to sync node with API server" Aug 05 16:21:25.179681 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.179680 2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 05 16:21:25.179740 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.179699 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 05 16:21:25.179740 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.179713 2573 kubelet.go:398] "Adding apiserver pod source" Aug 05 16:21:25.179740 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.179725 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 05 16:21:25.180501 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.180150 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 05 16:21:25.180501 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.180171 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 05 16:21:25.182320 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.182302 2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 05 16:21:25.183809 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.183794 2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 05 16:21:25.183867 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.183828 2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 05 16:21:25.185797 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.185785 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 05 16:21:25.185840 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.185803 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 05 16:21:25.185840 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.185810 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 05 16:21:25.185840 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.185816 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 05 16:21:25.185840 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.185826 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 05 16:21:25.185840 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.185832 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 05 16:21:25.185840 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.185840 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 05 16:21:25.186018 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.185847 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 05 16:21:25.186018 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.185854 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 05 16:21:25.186018 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.185860 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 05 16:21:25.186018 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.185869 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 05 16:21:25.186018 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.185875 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 05 16:21:25.186018 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.185899 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 05 16:21:25.186018 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.185905 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 05 16:21:25.189219 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.189201 2573 server.go:1268] "Started kubelet" Aug 05 16:21:25.189336 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.189277 2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 05 16:21:25.189397 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.189302 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 05 16:21:25.189464 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.189400 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 05 16:21:25.190118 ip-10-0-132-37 systemd[1]: Started Kubernetes Kubelet. Aug 05 16:21:25.190890 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.190689 2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 05 16:21:25.191264 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.191148 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-37.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 05 16:21:25.191313 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:25.191288 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-37.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 05 16:21:25.192202 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:25.192173 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 05 16:21:25.193917 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.193899 2573 server.go:310] "Adding debug handlers to kubelet server" Aug 05 16:21:25.195174 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:25.195152 2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 05 16:21:25.199547 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.199017 2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 05 16:21:25.199547 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.199028 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 05 16:21:25.199547 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:25.199147 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-37.ec2.internal\" not found" Aug 05 16:21:25.199547 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.199188 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fzt7z" Aug 05 16:21:25.199813 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.199599 2573 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 05 16:21:25.199813 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.199615 2573 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 05 16:21:25.199813 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.199599 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 05 16:21:25.199813 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.199704 2573 reconstruct.go:97] "Volume reconstruction finished" Aug 05 16:21:25.199813 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.199713 2573 reconciler.go:29] "Reconciler: start to sync state" Aug 05 16:21:25.200096 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.200071 2573 factory.go:55] Registering systemd factory Aug 05 16:21:25.200096 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.200087 2573 factory.go:223] Registration of the systemd container factory successfully Aug 05 16:21:25.200231 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:25.199114 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-37.ec2.internal.18c8f53acf6a7640 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-37.ec2.internal,UID:ip-10-0-132-37.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-37.ec2.internal,},FirstTimestamp:2026-08-05 16:21:25.189170752 +0000 UTC m=+0.346016647,LastTimestamp:2026-08-05 16:21:25.189170752 +0000 UTC m=+0.346016647,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-37.ec2.internal,}" Aug 05 16:21:25.200328 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.200309 2573 factory.go:153] Registering CRI-O factory Aug 05 16:21:25.200435 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.200342 2573 factory.go:223] Registration of the crio container factory successfully Aug 05 16:21:25.200435 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.200418 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 05 16:21:25.200537 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.200445 2573 factory.go:103] Registering Raw factory Aug 05 16:21:25.200537 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.200461 2573 manager.go:1196] Started watching for new ooms in manager Aug 05 16:21:25.201379 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.201361 2573 manager.go:349] Starting recovery of all containers Aug 05 16:21:25.207115 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:25.207084 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Aug 05 16:21:25.207115 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:25.207088 2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-37.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Aug 05 16:21:25.207565 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.207550 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fzt7z" Aug 05 16:21:25.211678 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.211655 2573 manager.go:354] Recovery completed Aug 05 16:21:25.216535 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.216521 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 16:21:25.218318 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.218302 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-37.ec2.internal" event="NodeHasSufficientMemory" Aug 05 16:21:25.218406 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.218337 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-37.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 16:21:25.218406 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.218353 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-37.ec2.internal" event="NodeHasSufficientPID" Aug 05 16:21:25.219419 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.219406 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 16:21:25.219518 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.219410 2573 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 05 16:21:25.219518 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.219473 2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 05 16:21:25.219518 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.219489 2573 state_mem.go:36] "Initialized new in-memory state store" Aug 05 16:21:25.220931 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.220917 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-37.ec2.internal" event="NodeHasSufficientMemory" Aug 05 16:21:25.221001 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.220948 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-37.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 16:21:25.221001 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.220964 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-37.ec2.internal" event="NodeHasSufficientPID" Aug 05 16:21:25.221103 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.221049 2573 policy_none.go:49] "None policy: Start" Aug 05 16:21:25.221411 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:25.221347 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-37.ec2.internal.18c8f53ad127390c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-37.ec2.internal,UID:ip-10-0-132-37.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-37.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-37.ec2.internal,},FirstTimestamp:2026-08-05 16:21:25.218318604 +0000 UTC m=+0.375164504,LastTimestamp:2026-08-05 16:21:25.218318604 +0000 UTC m=+0.375164504,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-37.ec2.internal,}" Aug 05 16:21:25.221886 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.221873 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 05 16:21:25.221935 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.221895 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 05 16:21:25.223119 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.223106 2573 policy_none.go:47] "Start" Aug 05 16:21:25.262240 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.262042 2573 manager.go:337] "Starting Device Plugin manager" Aug 05 16:21:25.262341 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:25.262270 2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 05 16:21:25.262341 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.262286 2573 server.go:87] "Starting device plugin registration server" Aug 05 16:21:25.262548 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.262536 2573 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 05 16:21:25.262621 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.262549 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 05 16:21:25.262701 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.262677 2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 05 16:21:25.262793 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.262781 2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 05 16:21:25.262842 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.262795 2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 05 16:21:25.263342 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:25.263321 2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 05 16:21:25.263436 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:25.263375 2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-37.ec2.internal\" not found" Aug 05 16:21:25.296366 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.296335 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 05 16:21:25.297631 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.297611 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 05 16:21:25.297734 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.297660 2573 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 05 16:21:25.297734 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.297688 2573 kubelet.go:2448] "Starting kubelet main sync loop" Aug 05 16:21:25.297734 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:25.297730 2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 05 16:21:25.300402 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.300378 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 16:21:25.365182 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.365118 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 16:21:25.366235 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.366214 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-37.ec2.internal" event="NodeHasSufficientMemory" Aug 05 16:21:25.366342 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.366250 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-37.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 16:21:25.366342 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.366266 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-37.ec2.internal" event="NodeHasSufficientPID" Aug 05 16:21:25.366342 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.366299 2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-37.ec2.internal" Aug 05 16:21:25.376184 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.376112 2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-37.ec2.internal" Aug 05 16:21:25.376184 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:25.376136 2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-37.ec2.internal\": node \"ip-10-0-132-37.ec2.internal\" not found" Aug 05 16:21:25.397841 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.397814 2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-37.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-37.ec2.internal"] Aug 05 16:21:25.397914 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.397882 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 16:21:25.398752 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:25.398730 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-37.ec2.internal\" not found" Aug 05 16:21:25.398870 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.398850 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-37.ec2.internal" event="NodeHasSufficientMemory" Aug 05 16:21:25.398922 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.398878 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-37.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 16:21:25.398922 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.398892 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-37.ec2.internal" event="NodeHasSufficientPID" Aug 05 16:21:25.400179 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.400165 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 16:21:25.400303 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.400286 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-37.ec2.internal" Aug 05 16:21:25.400357 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.400323 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 16:21:25.400968 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.400955 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-37.ec2.internal" event="NodeHasSufficientMemory" Aug 05 16:21:25.400968 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.400964 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-37.ec2.internal" event="NodeHasSufficientMemory" Aug 05 16:21:25.401085 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.400986 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-37.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 16:21:25.401085 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.400999 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-37.ec2.internal" event="NodeHasSufficientPID" Aug 05 16:21:25.401085 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.400985 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-37.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 16:21:25.401085 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.401075 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-37.ec2.internal" event="NodeHasSufficientPID" Aug 05 16:21:25.402183 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.402165 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-37.ec2.internal" Aug 05 16:21:25.402276 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.402193 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 16:21:25.403012 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.402995 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-37.ec2.internal" event="NodeHasSufficientMemory" Aug 05 16:21:25.403117 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.403020 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-37.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 16:21:25.403117 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.403036 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-37.ec2.internal" event="NodeHasSufficientPID" Aug 05 16:21:25.428236 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:25.428211 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-37.ec2.internal\" not found" node="ip-10-0-132-37.ec2.internal" Aug 05 16:21:25.433047 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:25.433031 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-37.ec2.internal\" not found" node="ip-10-0-132-37.ec2.internal" Aug 05 16:21:25.499645 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:25.499601 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-37.ec2.internal\" not found" Aug 05 16:21:25.500894 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.500874 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4408f2cc25089e2ed0e44bb381e90c02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-37.ec2.internal\" (UID: \"4408f2cc25089e2ed0e44bb381e90c02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-37.ec2.internal" Aug 05 16:21:25.500966 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.500905 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4408f2cc25089e2ed0e44bb381e90c02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-37.ec2.internal\" (UID: \"4408f2cc25089e2ed0e44bb381e90c02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-37.ec2.internal" Aug 05 16:21:25.500966 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.500934 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14d058665423739b2594851afe44a8fc-config\") pod \"kube-apiserver-proxy-ip-10-0-132-37.ec2.internal\" (UID: \"14d058665423739b2594851afe44a8fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-37.ec2.internal" Aug 05 16:21:25.600099 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:25.600050 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-37.ec2.internal\" not found" Aug 05 16:21:25.601317 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.601291 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4408f2cc25089e2ed0e44bb381e90c02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-37.ec2.internal\" (UID: \"4408f2cc25089e2ed0e44bb381e90c02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-37.ec2.internal" Aug 05 16:21:25.601429 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.601330 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4408f2cc25089e2ed0e44bb381e90c02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-37.ec2.internal\" (UID: \"4408f2cc25089e2ed0e44bb381e90c02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-37.ec2.internal" Aug 05 16:21:25.601429 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.601349 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14d058665423739b2594851afe44a8fc-config\") pod \"kube-apiserver-proxy-ip-10-0-132-37.ec2.internal\" (UID: \"14d058665423739b2594851afe44a8fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-37.ec2.internal" Aug 05 16:21:25.601429 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.601417 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14d058665423739b2594851afe44a8fc-config\") pod \"kube-apiserver-proxy-ip-10-0-132-37.ec2.internal\" (UID: \"14d058665423739b2594851afe44a8fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-37.ec2.internal" Aug 05 16:21:25.601547 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.601445 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4408f2cc25089e2ed0e44bb381e90c02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-37.ec2.internal\" (UID: \"4408f2cc25089e2ed0e44bb381e90c02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-37.ec2.internal" Aug 05 16:21:25.601547 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.601464 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4408f2cc25089e2ed0e44bb381e90c02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-37.ec2.internal\" (UID: \"4408f2cc25089e2ed0e44bb381e90c02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-37.ec2.internal" Aug 05 16:21:25.701125 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:25.701047 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-37.ec2.internal\" not found" Aug 05 16:21:25.732700 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.732672 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-37.ec2.internal" Aug 05 16:21:25.736045 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:25.736020 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-37.ec2.internal" Aug 05 16:21:25.801600 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:25.801545 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-37.ec2.internal\" not found" Aug 05 16:21:25.902524 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:25.902494 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-37.ec2.internal\" not found" Aug 05 16:21:26.003952 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:26.003881 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-37.ec2.internal\" not found" Aug 05 16:21:26.026778 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:26.026743 2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 16:21:26.104190 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:26.104144 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-37.ec2.internal\" not found" Aug 05 16:21:26.141138 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:26.141097 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 05 16:21:26.141621 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:26.141302 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 05 16:21:26.141621 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:26.141308 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 05 16:21:26.204295 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:26.204253 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-37.ec2.internal\" not found" Aug 05 16:21:26.212704 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:26.212660 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-04 16:16:25 +0000 UTC" deadline="2028-03-22 06:24:48.447478068 +0000 UTC" Aug 05 16:21:26.212704 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:26.212697 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14270h3m22.234785675s" Aug 05 16:21:26.304574 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:26.304485 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-37.ec2.internal\" not found" Aug 05 16:21:26.405013 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:26.404981 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-37.ec2.internal\" not found" Aug 05 16:21:26.505666 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:26.505629 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-37.ec2.internal\" not found" Aug 05 16:21:26.539631 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:26.539577 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4408f2cc25089e2ed0e44bb381e90c02.slice/crio-756935f4a4722a0d4622b5dbf80259ca2d9818d0ecd42a9db737d5635d3670ff WatchSource:0}: Error finding container 756935f4a4722a0d4622b5dbf80259ca2d9818d0ecd42a9db737d5635d3670ff: Status 404 returned error can't find the container with id 756935f4a4722a0d4622b5dbf80259ca2d9818d0ecd42a9db737d5635d3670ff Aug 05 16:21:26.539920 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:26.539900 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d058665423739b2594851afe44a8fc.slice/crio-a8bdc847145b20128dfb8fc59958f7c1526a2610e5c361cf27e2032cc1df754c WatchSource:0}: Error finding container a8bdc847145b20128dfb8fc59958f7c1526a2610e5c361cf27e2032cc1df754c: Status 404 returned error can't find the container with id a8bdc847145b20128dfb8fc59958f7c1526a2610e5c361cf27e2032cc1df754c Aug 05 16:21:26.544670 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:26.544652 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 05 16:21:26.605779 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:26.605686 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-37.ec2.internal\" not found" Aug 05 16:21:26.706091 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:26.706052 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-37.ec2.internal\" not found" Aug 05 16:21:26.741530 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:26.741506 2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 16:21:26.786399 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:26.786351 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 16:21:26.799805 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:26.799767 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-37.ec2.internal" Aug 05 16:21:26.813904 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:26.813871 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 05 16:21:26.814999 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:26.814982 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-37.ec2.internal" Aug 05 16:21:26.824752 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:26.824726 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 05 16:21:27.183714 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.183422 2573 apiserver.go:52] "Watching apiserver" Aug 05 16:21:27.191886 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.191828 2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 05 16:21:27.192400 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.192373 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-88qn5","openshift-network-diagnostics/network-check-target-pwx27","openshift-ovn-kubernetes/ovnkube-node-jjr22","kube-system/konnectivity-agent-kpd4q","kube-system/kube-apiserver-proxy-ip-10-0-132-37.ec2.internal","openshift-cluster-node-tuning-operator/tuned-xrv5x","openshift-dns/node-resolver-vbh4x","openshift-image-registry/node-ca-bmvw9","openshift-network-operator/iptables-alerter-8kt5n","kube-system/global-pull-secret-syncer-qjdbg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-37.ec2.internal","openshift-multus/multus-additional-cni-plugins-7kkz9","openshift-multus/multus-pfl76"] Aug 05 16:21:27.194946 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.194384 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vbh4x" Aug 05 16:21:27.197084 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.196349 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st" Aug 05 16:21:27.197427 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.197397 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 05 16:21:27.197713 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.197696 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jf7qf\"" Aug 05 16:21:27.198214 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.198195 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 05 16:21:27.200315 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.200292 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 05 16:21:27.200522 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.200504 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-txbvl\"" Aug 05 16:21:27.200769 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.200744 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 05 16:21:27.201053 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.201029 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 05 16:21:27.201188 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.201037 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 05 16:21:27.203904 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.203882 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.206475 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.206455 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kpd4q" Aug 05 16:21:27.207730 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.207706 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.208678 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.208657 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 05 16:21:27.208895 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.208875 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 05 16:21:27.208973 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.208920 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kzc92\"" Aug 05 16:21:27.209289 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.209222 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 05 16:21:27.209421 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.209385 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 05 16:21:27.209532 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.209460 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 05 16:21:27.209532 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.209464 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 05 16:21:27.209532 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.209497 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/621aa7b0-79b4-4b5a-a8bf-447ebe0a2075-etc-selinux\") pod \"aws-ebs-csi-driver-node-bb4st\" (UID: \"621aa7b0-79b4-4b5a-a8bf-447ebe0a2075\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st" Aug 05 16:21:27.209723 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.209534 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-var-lib-openvswitch\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.209723 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.209562 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-host-cni-bin\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.209723 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.209588 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/119a5576-d632-47e3-8692-4a4a0e1b7191-ovn-node-metrics-cert\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.209723 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.209612 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/621aa7b0-79b4-4b5a-a8bf-447ebe0a2075-socket-dir\") pod \"aws-ebs-csi-driver-node-bb4st\" (UID: \"621aa7b0-79b4-4b5a-a8bf-447ebe0a2075\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st" Aug 05 16:21:27.209723 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.209658 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-node-log\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.209723 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.209698 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dznmp\" (UniqueName: \"kubernetes.io/projected/fe314431-1429-4319-b3a0-0c7287575cef-kube-api-access-dznmp\") pod \"node-resolver-vbh4x\" (UID: \"fe314431-1429-4319-b3a0-0c7287575cef\") " pod="openshift-dns/node-resolver-vbh4x" Aug 05 16:21:27.210105 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.209729 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-run-openvswitch\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.210105 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.209756 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.210105 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.209793 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-log-socket\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.210105 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.209837 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 05 16:21:27.210105 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.209844 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-host-run-ovn-kubernetes\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.210105 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.209873 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-host-cni-netd\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.210105 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.209901 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxhgm\" (UniqueName: \"kubernetes.io/projected/621aa7b0-79b4-4b5a-a8bf-447ebe0a2075-kube-api-access-dxhgm\") pod \"aws-ebs-csi-driver-node-bb4st\" (UID: \"621aa7b0-79b4-4b5a-a8bf-447ebe0a2075\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st" Aug 05 16:21:27.210105 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.209928 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-run-ovn\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.210105 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.209954 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-host-kubelet\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.210105 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.209977 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/119a5576-d632-47e3-8692-4a4a0e1b7191-ovnkube-config\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.210105 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.210007 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-host-slash\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.210105 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.210048 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-run-systemd\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.210105 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.210069 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.210105 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.210095 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fe314431-1429-4319-b3a0-0c7287575cef-hosts-file\") pod \"node-resolver-vbh4x\" (UID: \"fe314431-1429-4319-b3a0-0c7287575cef\") " pod="openshift-dns/node-resolver-vbh4x" Aug 05 16:21:27.210105 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.210109 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 05 16:21:27.210105 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.210120 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/621aa7b0-79b4-4b5a-a8bf-447ebe0a2075-device-dir\") pod \"aws-ebs-csi-driver-node-bb4st\" (UID: \"621aa7b0-79b4-4b5a-a8bf-447ebe0a2075\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st" Aug 05 16:21:27.211102 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.210160 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/621aa7b0-79b4-4b5a-a8bf-447ebe0a2075-sys-fs\") pod \"aws-ebs-csi-driver-node-bb4st\" (UID: \"621aa7b0-79b4-4b5a-a8bf-447ebe0a2075\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st" Aug 05 16:21:27.211102 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.210172 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-j6hpc\"" Aug 05 16:21:27.211102 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.210184 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-host-run-netns\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.211102 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.210209 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-etc-openvswitch\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.211102 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.210235 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/119a5576-d632-47e3-8692-4a4a0e1b7191-ovnkube-script-lib\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.211102 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.210259 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/621aa7b0-79b4-4b5a-a8bf-447ebe0a2075-registration-dir\") pod \"aws-ebs-csi-driver-node-bb4st\" (UID: \"621aa7b0-79b4-4b5a-a8bf-447ebe0a2075\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st" Aug 05 16:21:27.211102 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.210303 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-systemd-units\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.211102 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.210335 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/119a5576-d632-47e3-8692-4a4a0e1b7191-env-overrides\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.211102 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.210399 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j72k\" (UniqueName: \"kubernetes.io/projected/119a5576-d632-47e3-8692-4a4a0e1b7191-kube-api-access-8j72k\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.220049 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.220025 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bcvpm\"" Aug 05 16:21:27.222268 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.220360 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 05 16:21:27.222268 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.220598 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 05 16:21:27.222268 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.220732 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 05 16:21:27.222268 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.220811 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 05 16:21:27.222268 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.221048 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p64ww\"" Aug 05 16:21:27.222268 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.220360 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 05 16:21:27.222268 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.210432 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe314431-1429-4319-b3a0-0c7287575cef-tmp-dir\") pod \"node-resolver-vbh4x\" (UID: \"fe314431-1429-4319-b3a0-0c7287575cef\") " pod="openshift-dns/node-resolver-vbh4x" Aug 05 16:21:27.222268 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.222100 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/621aa7b0-79b4-4b5a-a8bf-447ebe0a2075-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bb4st\" (UID: \"621aa7b0-79b4-4b5a-a8bf-447ebe0a2075\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st" Aug 05 16:21:27.222268 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.222141 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8kt5n" Aug 05 16:21:27.224402 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.223330 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 05 16:21:27.224511 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.224472 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7kkz9" Aug 05 16:21:27.224832 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.224812 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 05 16:21:27.228081 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.228062 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:21:27.228175 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:27.228130 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwx27" podUID="e303e40d-9669-4230-a1e1-c75ea3a2c4df" Aug 05 16:21:27.228830 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.228810 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 05 16:21:27.229208 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.229189 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 05 16:21:27.229380 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.229359 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 05 16:21:27.229896 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.229638 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:27.229896 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.229719 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 05 16:21:27.229896 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:27.229732 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjdbg" podUID="39d233df-9262-45b2-8065-0d44a7ee9e12" Aug 05 16:21:27.229896 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.229759 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 05 16:21:27.229896 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.229838 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2glc4\"" Aug 05 16:21:27.230210 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.229977 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6r27v\"" Aug 05 16:21:27.231433 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.231415 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:27.231530 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:27.231497 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88qn5" podUID="aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d" Aug 05 16:21:27.232609 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.232592 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bmvw9" Aug 05 16:21:27.236312 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.236291 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 05 16:21:27.236647 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.236521 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 05 16:21:27.236753 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.236668 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ccjk8\"" Aug 05 16:21:27.237155 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.237129 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 05 16:21:27.253075 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.253043 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ktspb" Aug 05 16:21:27.261616 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.261595 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ktspb" Aug 05 16:21:27.269956 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.269930 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 16:21:27.302280 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.302252 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 05 16:21:27.306117 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.306062 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-37.ec2.internal" event={"ID":"14d058665423739b2594851afe44a8fc","Type":"ContainerStarted","Data":"a8bdc847145b20128dfb8fc59958f7c1526a2610e5c361cf27e2032cc1df754c"} Aug 05 16:21:27.308299 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.308274 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-37.ec2.internal" event={"ID":"4408f2cc25089e2ed0e44bb381e90c02","Type":"ContainerStarted","Data":"756935f4a4722a0d4622b5dbf80259ca2d9818d0ecd42a9db737d5635d3670ff"} Aug 05 16:21:27.322689 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.322661 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-host-kubelet\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.322797 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.322703 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/119a5576-d632-47e3-8692-4a4a0e1b7191-ovnkube-config\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.322797 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.322740 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-system-cni-dir\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.322797 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.322766 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-etc-sysctl-d\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.322797 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.322791 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-etc-systemd\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.322939 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.322817 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-run\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.322939 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.322841 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-host\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.322939 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.322865 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvgjt\" (UniqueName: \"kubernetes.io/projected/93035306-7fa8-40fa-9517-2992e9340f56-kube-api-access-xvgjt\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.322939 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.322895 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fe314431-1429-4319-b3a0-0c7287575cef-hosts-file\") pod \"node-resolver-vbh4x\" (UID: \"fe314431-1429-4319-b3a0-0c7287575cef\") " pod="openshift-dns/node-resolver-vbh4x" Aug 05 16:21:27.322939 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.322919 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/333a43c6-e0d3-42ab-a596-3670a69c8d5d-agent-certs\") pod \"konnectivity-agent-kpd4q\" (UID: \"333a43c6-e0d3-42ab-a596-3670a69c8d5d\") " pod="kube-system/konnectivity-agent-kpd4q" Aug 05 16:21:27.323089 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.322945 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/39d233df-9262-45b2-8065-0d44a7ee9e12-kubelet-config\") pod \"global-pull-secret-syncer-qjdbg\" (UID: \"39d233df-9262-45b2-8065-0d44a7ee9e12\") " pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:27.323089 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.322970 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d-metrics-certs\") pod \"network-metrics-daemon-88qn5\" (UID: \"aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d\") " pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:27.323089 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.322996 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62sl8\" (UniqueName: \"kubernetes.io/projected/aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d-kube-api-access-62sl8\") pod \"network-metrics-daemon-88qn5\" (UID: \"aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d\") " pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:27.323089 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323033 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6bnc\" (UniqueName: \"kubernetes.io/projected/f2a0163b-3977-4a52-8c23-6139f01a9b2d-kube-api-access-z6bnc\") pod \"node-ca-bmvw9\" (UID: \"f2a0163b-3977-4a52-8c23-6139f01a9b2d\") " pod="openshift-image-registry/node-ca-bmvw9" Aug 05 16:21:27.323089 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323060 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/119a5576-d632-47e3-8692-4a4a0e1b7191-ovnkube-script-lib\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.323240 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323087 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/621aa7b0-79b4-4b5a-a8bf-447ebe0a2075-registration-dir\") pod \"aws-ebs-csi-driver-node-bb4st\" (UID: \"621aa7b0-79b4-4b5a-a8bf-447ebe0a2075\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st" Aug 05 16:21:27.323240 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323111 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-etc-kubernetes\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.323240 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323140 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/63118331-baa7-4178-9751-3403362e5224-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7kkz9\" (UID: \"63118331-baa7-4178-9751-3403362e5224\") " pod="openshift-multus/multus-additional-cni-plugins-7kkz9" Aug 05 16:21:27.323240 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323166 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j72k\" (UniqueName: \"kubernetes.io/projected/119a5576-d632-47e3-8692-4a4a0e1b7191-kube-api-access-8j72k\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.323240 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323191 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe314431-1429-4319-b3a0-0c7287575cef-tmp-dir\") pod \"node-resolver-vbh4x\" (UID: \"fe314431-1429-4319-b3a0-0c7287575cef\") " pod="openshift-dns/node-resolver-vbh4x" Aug 05 16:21:27.323240 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323217 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/333a43c6-e0d3-42ab-a596-3670a69c8d5d-konnectivity-ca\") pod \"konnectivity-agent-kpd4q\" (UID: \"333a43c6-e0d3-42ab-a596-3670a69c8d5d\") " pod="kube-system/konnectivity-agent-kpd4q" Aug 05 16:21:27.324900 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323253 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/119a5576-d632-47e3-8692-4a4a0e1b7191-ovn-node-metrics-cert\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.324900 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323277 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-host-run-netns\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.324900 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323302 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-host-var-lib-cni-multus\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.324900 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323330 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-etc-sysconfig\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.324900 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323374 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-etc-kubernetes\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.324900 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323400 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63118331-baa7-4178-9751-3403362e5224-cnibin\") pod \"multus-additional-cni-plugins-7kkz9\" (UID: \"63118331-baa7-4178-9751-3403362e5224\") " pod="openshift-multus/multus-additional-cni-plugins-7kkz9" Aug 05 16:21:27.324900 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323431 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ts9s\" (UniqueName: \"kubernetes.io/projected/63118331-baa7-4178-9751-3403362e5224-kube-api-access-2ts9s\") pod \"multus-additional-cni-plugins-7kkz9\" (UID: \"63118331-baa7-4178-9751-3403362e5224\") " pod="openshift-multus/multus-additional-cni-plugins-7kkz9" Aug 05 16:21:27.324900 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323474 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dznmp\" (UniqueName: \"kubernetes.io/projected/fe314431-1429-4319-b3a0-0c7287575cef-kube-api-access-dznmp\") pod \"node-resolver-vbh4x\" (UID: \"fe314431-1429-4319-b3a0-0c7287575cef\") " pod="openshift-dns/node-resolver-vbh4x" Aug 05 16:21:27.324900 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323498 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-host-kubelet\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.324900 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323609 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fe314431-1429-4319-b3a0-0c7287575cef-hosts-file\") pod \"node-resolver-vbh4x\" (UID: \"fe314431-1429-4319-b3a0-0c7287575cef\") " pod="openshift-dns/node-resolver-vbh4x" Aug 05 16:21:27.324900 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323501 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/39d233df-9262-45b2-8065-0d44a7ee9e12-dbus\") pod \"global-pull-secret-syncer-qjdbg\" (UID: \"39d233df-9262-45b2-8065-0d44a7ee9e12\") " pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:27.324900 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323690 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.324900 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323723 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-cnibin\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.324900 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323749 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/866c70f0-ff16-4ca9-b8db-730393bff7f4-multus-daemon-config\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.324900 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323773 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-lib-modules\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.324900 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323801 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65zvj\" (UniqueName: \"kubernetes.io/projected/5240cc7e-7437-476c-a63c-fd147054d87c-kube-api-access-65zvj\") pod \"iptables-alerter-8kt5n\" (UID: \"5240cc7e-7437-476c-a63c-fd147054d87c\") " pod="openshift-network-operator/iptables-alerter-8kt5n" Aug 05 16:21:27.324900 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323830 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-log-socket\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.325923 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323857 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-host-run-ovn-kubernetes\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.325923 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323885 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-host-cni-netd\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.325923 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323912 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-multus-socket-dir-parent\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.325923 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323947 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/93035306-7fa8-40fa-9517-2992e9340f56-etc-tuned\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.325923 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.323974 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-run-ovn\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.325923 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324002 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-host-var-lib-kubelet\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.325923 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324028 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-etc-sysctl-conf\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.325923 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324056 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-var-lib-kubelet\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.325923 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324082 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5240cc7e-7437-476c-a63c-fd147054d87c-host-slash\") pod \"iptables-alerter-8kt5n\" (UID: \"5240cc7e-7437-476c-a63c-fd147054d87c\") " pod="openshift-network-operator/iptables-alerter-8kt5n" Aug 05 16:21:27.325923 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324108 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-host-slash\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.325923 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324134 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-run-systemd\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.325923 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324146 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/119a5576-d632-47e3-8692-4a4a0e1b7191-ovnkube-config\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.325923 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324160 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/621aa7b0-79b4-4b5a-a8bf-447ebe0a2075-device-dir\") pod \"aws-ebs-csi-driver-node-bb4st\" (UID: \"621aa7b0-79b4-4b5a-a8bf-447ebe0a2075\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st" Aug 05 16:21:27.325923 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324191 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/621aa7b0-79b4-4b5a-a8bf-447ebe0a2075-sys-fs\") pod \"aws-ebs-csi-driver-node-bb4st\" (UID: \"621aa7b0-79b4-4b5a-a8bf-447ebe0a2075\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st" Aug 05 16:21:27.325923 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324218 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5240cc7e-7437-476c-a63c-fd147054d87c-iptables-alerter-script\") pod \"iptables-alerter-8kt5n\" (UID: \"5240cc7e-7437-476c-a63c-fd147054d87c\") " pod="openshift-network-operator/iptables-alerter-8kt5n" Aug 05 16:21:27.325923 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324245 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63118331-baa7-4178-9751-3403362e5224-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7kkz9\" (UID: \"63118331-baa7-4178-9751-3403362e5224\") " pod="openshift-multus/multus-additional-cni-plugins-7kkz9" Aug 05 16:21:27.325923 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324290 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-host-run-netns\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.326806 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324322 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-etc-openvswitch\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.326806 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324347 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x5w8\" (UniqueName: \"kubernetes.io/projected/e303e40d-9669-4230-a1e1-c75ea3a2c4df-kube-api-access-8x5w8\") pod \"network-check-target-pwx27\" (UID: \"e303e40d-9669-4230-a1e1-c75ea3a2c4df\") " pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:21:27.326806 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324418 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.326806 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324472 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-run-systemd\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.326806 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324488 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/621aa7b0-79b4-4b5a-a8bf-447ebe0a2075-registration-dir\") pod \"aws-ebs-csi-driver-node-bb4st\" (UID: \"621aa7b0-79b4-4b5a-a8bf-447ebe0a2075\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st" Aug 05 16:21:27.326806 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324535 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-systemd-units\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.326806 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324541 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/621aa7b0-79b4-4b5a-a8bf-447ebe0a2075-device-dir\") pod \"aws-ebs-csi-driver-node-bb4st\" (UID: \"621aa7b0-79b4-4b5a-a8bf-447ebe0a2075\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st" Aug 05 16:21:27.326806 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324568 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/119a5576-d632-47e3-8692-4a4a0e1b7191-env-overrides\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.326806 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324581 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-log-socket\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.326806 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324595 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/621aa7b0-79b4-4b5a-a8bf-447ebe0a2075-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bb4st\" (UID: \"621aa7b0-79b4-4b5a-a8bf-447ebe0a2075\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st" Aug 05 16:21:27.326806 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324621 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-host-run-ovn-kubernetes\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.326806 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324632 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/621aa7b0-79b4-4b5a-a8bf-447ebe0a2075-etc-selinux\") pod \"aws-ebs-csi-driver-node-bb4st\" (UID: \"621aa7b0-79b4-4b5a-a8bf-447ebe0a2075\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st" Aug 05 16:21:27.326806 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324712 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-systemd-units\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.326806 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324491 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-run-ovn\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.326806 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324751 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-host-slash\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.326806 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324810 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-host-var-lib-cni-bin\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.326806 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324844 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63118331-baa7-4178-9751-3403362e5224-system-cni-dir\") pod \"multus-additional-cni-plugins-7kkz9\" (UID: \"63118331-baa7-4178-9751-3403362e5224\") " pod="openshift-multus/multus-additional-cni-plugins-7kkz9" Aug 05 16:21:27.327730 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324872 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f2a0163b-3977-4a52-8c23-6139f01a9b2d-serviceca\") pod \"node-ca-bmvw9\" (UID: \"f2a0163b-3977-4a52-8c23-6139f01a9b2d\") " pod="openshift-image-registry/node-ca-bmvw9" Aug 05 16:21:27.327730 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324881 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/621aa7b0-79b4-4b5a-a8bf-447ebe0a2075-etc-selinux\") pod \"aws-ebs-csi-driver-node-bb4st\" (UID: \"621aa7b0-79b4-4b5a-a8bf-447ebe0a2075\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st" Aug 05 16:21:27.327730 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324906 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-var-lib-openvswitch\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.327730 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324917 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/621aa7b0-79b4-4b5a-a8bf-447ebe0a2075-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bb4st\" (UID: \"621aa7b0-79b4-4b5a-a8bf-447ebe0a2075\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st" Aug 05 16:21:27.327730 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324934 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-host-cni-bin\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.327730 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324954 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-host-cni-netd\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.327730 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.324959 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/621aa7b0-79b4-4b5a-a8bf-447ebe0a2075-socket-dir\") pod \"aws-ebs-csi-driver-node-bb4st\" (UID: \"621aa7b0-79b4-4b5a-a8bf-447ebe0a2075\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st" Aug 05 16:21:27.327730 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.325086 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/621aa7b0-79b4-4b5a-a8bf-447ebe0a2075-socket-dir\") pod \"aws-ebs-csi-driver-node-bb4st\" (UID: \"621aa7b0-79b4-4b5a-a8bf-447ebe0a2075\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st" Aug 05 16:21:27.327730 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.325125 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/621aa7b0-79b4-4b5a-a8bf-447ebe0a2075-sys-fs\") pod \"aws-ebs-csi-driver-node-bb4st\" (UID: \"621aa7b0-79b4-4b5a-a8bf-447ebe0a2075\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st" Aug 05 16:21:27.327730 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.325171 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-var-lib-openvswitch\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.327730 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.325212 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-host-cni-bin\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.327730 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.325233 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe314431-1429-4319-b3a0-0c7287575cef-tmp-dir\") pod \"node-resolver-vbh4x\" (UID: \"fe314431-1429-4319-b3a0-0c7287575cef\") " pod="openshift-dns/node-resolver-vbh4x" Aug 05 16:21:27.327730 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.325256 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-host-run-netns\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.327730 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.325281 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-sys\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.327730 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.325290 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-etc-openvswitch\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.327730 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.325325 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/63118331-baa7-4178-9751-3403362e5224-os-release\") pod \"multus-additional-cni-plugins-7kkz9\" (UID: \"63118331-baa7-4178-9751-3403362e5224\") " pod="openshift-multus/multus-additional-cni-plugins-7kkz9" Aug 05 16:21:27.327730 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.325358 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-node-log\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.328593 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.325384 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-hostroot\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.328593 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.325410 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-host-run-multus-certs\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.328593 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.325428 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-node-log\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.328593 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.325464 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhvvk\" (UniqueName: \"kubernetes.io/projected/866c70f0-ff16-4ca9-b8db-730393bff7f4-kube-api-access-vhvvk\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.328593 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.325517 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-run-openvswitch\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.328593 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.325559 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-multus-cni-dir\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.328593 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.325591 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-os-release\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.328593 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.325628 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-host-run-k8s-cni-cncf-io\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.328593 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.325656 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-multus-conf-dir\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.328593 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.325706 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxhgm\" (UniqueName: \"kubernetes.io/projected/621aa7b0-79b4-4b5a-a8bf-447ebe0a2075-kube-api-access-dxhgm\") pod \"aws-ebs-csi-driver-node-bb4st\" (UID: \"621aa7b0-79b4-4b5a-a8bf-447ebe0a2075\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st" Aug 05 16:21:27.328593 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.325722 2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 05 16:21:27.328593 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.325735 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/866c70f0-ff16-4ca9-b8db-730393bff7f4-cni-binary-copy\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.328593 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.325791 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-etc-modprobe-d\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.328593 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.325819 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93035306-7fa8-40fa-9517-2992e9340f56-tmp\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.328593 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.325846 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d233df-9262-45b2-8065-0d44a7ee9e12-original-pull-secret\") pod \"global-pull-secret-syncer-qjdbg\" (UID: \"39d233df-9262-45b2-8065-0d44a7ee9e12\") " pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:27.328593 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.325875 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63118331-baa7-4178-9751-3403362e5224-cni-binary-copy\") pod \"multus-additional-cni-plugins-7kkz9\" (UID: \"63118331-baa7-4178-9751-3403362e5224\") " pod="openshift-multus/multus-additional-cni-plugins-7kkz9" Aug 05 16:21:27.328593 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.325906 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/63118331-baa7-4178-9751-3403362e5224-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7kkz9\" (UID: \"63118331-baa7-4178-9751-3403362e5224\") " pod="openshift-multus/multus-additional-cni-plugins-7kkz9" Aug 05 16:21:27.329501 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.325932 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2a0163b-3977-4a52-8c23-6139f01a9b2d-host\") pod \"node-ca-bmvw9\" (UID: \"f2a0163b-3977-4a52-8c23-6139f01a9b2d\") " pod="openshift-image-registry/node-ca-bmvw9" Aug 05 16:21:27.329501 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.326014 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/119a5576-d632-47e3-8692-4a4a0e1b7191-env-overrides\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.329501 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.326035 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/119a5576-d632-47e3-8692-4a4a0e1b7191-ovnkube-script-lib\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.329501 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.326066 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/119a5576-d632-47e3-8692-4a4a0e1b7191-run-openvswitch\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.329501 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.329423 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/119a5576-d632-47e3-8692-4a4a0e1b7191-ovn-node-metrics-cert\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.340329 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.340243 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j72k\" (UniqueName: \"kubernetes.io/projected/119a5576-d632-47e3-8692-4a4a0e1b7191-kube-api-access-8j72k\") pod \"ovnkube-node-jjr22\" (UID: \"119a5576-d632-47e3-8692-4a4a0e1b7191\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.341881 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.341837 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxhgm\" (UniqueName: \"kubernetes.io/projected/621aa7b0-79b4-4b5a-a8bf-447ebe0a2075-kube-api-access-dxhgm\") pod \"aws-ebs-csi-driver-node-bb4st\" (UID: \"621aa7b0-79b4-4b5a-a8bf-447ebe0a2075\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st" Aug 05 16:21:27.342394 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.342369 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dznmp\" (UniqueName: \"kubernetes.io/projected/fe314431-1429-4319-b3a0-0c7287575cef-kube-api-access-dznmp\") pod \"node-resolver-vbh4x\" (UID: \"fe314431-1429-4319-b3a0-0c7287575cef\") " pod="openshift-dns/node-resolver-vbh4x" Aug 05 16:21:27.426343 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.426307 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-system-cni-dir\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.426543 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.426351 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-etc-sysctl-d\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.426543 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.426376 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-etc-systemd\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.426543 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.426411 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-run\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.426543 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.426432 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-host\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.426543 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.426454 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvgjt\" (UniqueName: \"kubernetes.io/projected/93035306-7fa8-40fa-9517-2992e9340f56-kube-api-access-xvgjt\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.426543 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.426459 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-system-cni-dir\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.426543 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.426483 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/333a43c6-e0d3-42ab-a596-3670a69c8d5d-agent-certs\") pod \"konnectivity-agent-kpd4q\" (UID: \"333a43c6-e0d3-42ab-a596-3670a69c8d5d\") " pod="kube-system/konnectivity-agent-kpd4q" Aug 05 16:21:27.426925 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.426695 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-etc-systemd\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.426925 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.426724 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-etc-sysctl-d\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.426925 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.426743 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/39d233df-9262-45b2-8065-0d44a7ee9e12-kubelet-config\") pod \"global-pull-secret-syncer-qjdbg\" (UID: \"39d233df-9262-45b2-8065-0d44a7ee9e12\") " pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:27.426925 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.426771 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-run\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.426925 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.426775 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d-metrics-certs\") pod \"network-metrics-daemon-88qn5\" (UID: \"aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d\") " pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:27.426925 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.426818 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62sl8\" (UniqueName: \"kubernetes.io/projected/aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d-kube-api-access-62sl8\") pod \"network-metrics-daemon-88qn5\" (UID: \"aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d\") " pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:27.426925 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.426849 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6bnc\" (UniqueName: \"kubernetes.io/projected/f2a0163b-3977-4a52-8c23-6139f01a9b2d-kube-api-access-z6bnc\") pod \"node-ca-bmvw9\" (UID: \"f2a0163b-3977-4a52-8c23-6139f01a9b2d\") " pod="openshift-image-registry/node-ca-bmvw9" Aug 05 16:21:27.426925 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:27.426860 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:21:27.426925 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.426856 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-host\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.426925 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.426913 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/39d233df-9262-45b2-8065-0d44a7ee9e12-kubelet-config\") pod \"global-pull-secret-syncer-qjdbg\" (UID: \"39d233df-9262-45b2-8065-0d44a7ee9e12\") " pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:27.427442 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427002 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-etc-kubernetes\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.427442 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:27.427056 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d-metrics-certs podName:aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d nodeName:}" failed. No retries permitted until 2026-08-05 16:21:27.927029477 +0000 UTC m=+3.083875380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d-metrics-certs") pod "network-metrics-daemon-88qn5" (UID: "aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:21:27.427442 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427055 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-etc-kubernetes\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.427442 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427131 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/63118331-baa7-4178-9751-3403362e5224-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7kkz9\" (UID: \"63118331-baa7-4178-9751-3403362e5224\") " pod="openshift-multus/multus-additional-cni-plugins-7kkz9" Aug 05 16:21:27.427442 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427167 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/333a43c6-e0d3-42ab-a596-3670a69c8d5d-konnectivity-ca\") pod \"konnectivity-agent-kpd4q\" (UID: \"333a43c6-e0d3-42ab-a596-3670a69c8d5d\") " pod="kube-system/konnectivity-agent-kpd4q" Aug 05 16:21:27.427442 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427202 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-host-run-netns\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.427442 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427233 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-host-var-lib-cni-multus\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.427442 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427259 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-etc-sysconfig\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.427442 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427284 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-etc-kubernetes\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.427442 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427308 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63118331-baa7-4178-9751-3403362e5224-cnibin\") pod \"multus-additional-cni-plugins-7kkz9\" (UID: \"63118331-baa7-4178-9751-3403362e5224\") " pod="openshift-multus/multus-additional-cni-plugins-7kkz9" Aug 05 16:21:27.427442 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427334 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ts9s\" (UniqueName: \"kubernetes.io/projected/63118331-baa7-4178-9751-3403362e5224-kube-api-access-2ts9s\") pod \"multus-additional-cni-plugins-7kkz9\" (UID: \"63118331-baa7-4178-9751-3403362e5224\") " pod="openshift-multus/multus-additional-cni-plugins-7kkz9" Aug 05 16:21:27.427442 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427359 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/39d233df-9262-45b2-8065-0d44a7ee9e12-dbus\") pod \"global-pull-secret-syncer-qjdbg\" (UID: \"39d233df-9262-45b2-8065-0d44a7ee9e12\") " pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:27.427442 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427411 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-cnibin\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.428136 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427476 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-etc-kubernetes\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.428136 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427508 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63118331-baa7-4178-9751-3403362e5224-cnibin\") pod \"multus-additional-cni-plugins-7kkz9\" (UID: \"63118331-baa7-4178-9751-3403362e5224\") " pod="openshift-multus/multus-additional-cni-plugins-7kkz9" Aug 05 16:21:27.428136 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427538 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-cnibin\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.428136 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427582 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-host-run-netns\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.428136 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427622 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-host-var-lib-cni-multus\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.428136 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427672 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-etc-sysconfig\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.428136 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427699 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/866c70f0-ff16-4ca9-b8db-730393bff7f4-multus-daemon-config\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.428136 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427725 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-lib-modules\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.428136 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427734 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/333a43c6-e0d3-42ab-a596-3670a69c8d5d-konnectivity-ca\") pod \"konnectivity-agent-kpd4q\" (UID: \"333a43c6-e0d3-42ab-a596-3670a69c8d5d\") " pod="kube-system/konnectivity-agent-kpd4q" Aug 05 16:21:27.428136 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427752 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65zvj\" (UniqueName: \"kubernetes.io/projected/5240cc7e-7437-476c-a63c-fd147054d87c-kube-api-access-65zvj\") pod \"iptables-alerter-8kt5n\" (UID: \"5240cc7e-7437-476c-a63c-fd147054d87c\") " pod="openshift-network-operator/iptables-alerter-8kt5n" Aug 05 16:21:27.428136 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427734 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/63118331-baa7-4178-9751-3403362e5224-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7kkz9\" (UID: \"63118331-baa7-4178-9751-3403362e5224\") " pod="openshift-multus/multus-additional-cni-plugins-7kkz9" Aug 05 16:21:27.428136 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427787 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-multus-socket-dir-parent\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.428136 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427812 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/93035306-7fa8-40fa-9517-2992e9340f56-etc-tuned\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.428136 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427838 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-host-var-lib-kubelet\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.428136 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427869 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/39d233df-9262-45b2-8065-0d44a7ee9e12-dbus\") pod \"global-pull-secret-syncer-qjdbg\" (UID: \"39d233df-9262-45b2-8065-0d44a7ee9e12\") " pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:27.428136 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427874 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-lib-modules\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.428136 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427863 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-etc-sysctl-conf\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.428136 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427915 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-var-lib-kubelet\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.429063 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427945 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-multus-socket-dir-parent\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.429063 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427947 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5240cc7e-7437-476c-a63c-fd147054d87c-host-slash\") pod \"iptables-alerter-8kt5n\" (UID: \"5240cc7e-7437-476c-a63c-fd147054d87c\") " pod="openshift-network-operator/iptables-alerter-8kt5n" Aug 05 16:21:27.429063 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427976 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5240cc7e-7437-476c-a63c-fd147054d87c-host-slash\") pod \"iptables-alerter-8kt5n\" (UID: \"5240cc7e-7437-476c-a63c-fd147054d87c\") " pod="openshift-network-operator/iptables-alerter-8kt5n" Aug 05 16:21:27.429063 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.427987 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5240cc7e-7437-476c-a63c-fd147054d87c-iptables-alerter-script\") pod \"iptables-alerter-8kt5n\" (UID: \"5240cc7e-7437-476c-a63c-fd147054d87c\") " pod="openshift-network-operator/iptables-alerter-8kt5n" Aug 05 16:21:27.429063 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428009 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-etc-sysctl-conf\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.429063 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428016 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63118331-baa7-4178-9751-3403362e5224-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7kkz9\" (UID: \"63118331-baa7-4178-9751-3403362e5224\") " pod="openshift-multus/multus-additional-cni-plugins-7kkz9" Aug 05 16:21:27.429063 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428028 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-var-lib-kubelet\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.429063 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428050 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x5w8\" (UniqueName: \"kubernetes.io/projected/e303e40d-9669-4230-a1e1-c75ea3a2c4df-kube-api-access-8x5w8\") pod \"network-check-target-pwx27\" (UID: \"e303e40d-9669-4230-a1e1-c75ea3a2c4df\") " pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:21:27.429063 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428069 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-host-var-lib-kubelet\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.429063 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428082 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-host-var-lib-cni-bin\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.429063 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428137 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63118331-baa7-4178-9751-3403362e5224-system-cni-dir\") pod \"multus-additional-cni-plugins-7kkz9\" (UID: \"63118331-baa7-4178-9751-3403362e5224\") " pod="openshift-multus/multus-additional-cni-plugins-7kkz9" Aug 05 16:21:27.429063 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428165 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f2a0163b-3977-4a52-8c23-6139f01a9b2d-serviceca\") pod \"node-ca-bmvw9\" (UID: \"f2a0163b-3977-4a52-8c23-6139f01a9b2d\") " pod="openshift-image-registry/node-ca-bmvw9" Aug 05 16:21:27.429063 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428196 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-sys\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.429063 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428223 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/63118331-baa7-4178-9751-3403362e5224-os-release\") pod \"multus-additional-cni-plugins-7kkz9\" (UID: \"63118331-baa7-4178-9751-3403362e5224\") " pod="openshift-multus/multus-additional-cni-plugins-7kkz9" Aug 05 16:21:27.429063 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428251 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-hostroot\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.429063 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428280 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-host-run-multus-certs\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.429063 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428301 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhvvk\" (UniqueName: \"kubernetes.io/projected/866c70f0-ff16-4ca9-b8db-730393bff7f4-kube-api-access-vhvvk\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.429944 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428306 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/866c70f0-ff16-4ca9-b8db-730393bff7f4-multus-daemon-config\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.429944 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428327 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-multus-cni-dir\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.429944 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428352 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-os-release\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.429944 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428378 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-host-run-k8s-cni-cncf-io\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.429944 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428393 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-sys\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.429944 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428401 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-multus-conf-dir\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.429944 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428434 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/866c70f0-ff16-4ca9-b8db-730393bff7f4-cni-binary-copy\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.429944 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428468 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-host-var-lib-cni-bin\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.429944 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428475 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-etc-modprobe-d\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.429944 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428500 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93035306-7fa8-40fa-9517-2992e9340f56-tmp\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.429944 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428519 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63118331-baa7-4178-9751-3403362e5224-system-cni-dir\") pod \"multus-additional-cni-plugins-7kkz9\" (UID: \"63118331-baa7-4178-9751-3403362e5224\") " pod="openshift-multus/multus-additional-cni-plugins-7kkz9" Aug 05 16:21:27.429944 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428524 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d233df-9262-45b2-8065-0d44a7ee9e12-original-pull-secret\") pod \"global-pull-secret-syncer-qjdbg\" (UID: \"39d233df-9262-45b2-8065-0d44a7ee9e12\") " pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:27.429944 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428556 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63118331-baa7-4178-9751-3403362e5224-cni-binary-copy\") pod \"multus-additional-cni-plugins-7kkz9\" (UID: \"63118331-baa7-4178-9751-3403362e5224\") " pod="openshift-multus/multus-additional-cni-plugins-7kkz9" Aug 05 16:21:27.429944 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428581 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/63118331-baa7-4178-9751-3403362e5224-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7kkz9\" (UID: \"63118331-baa7-4178-9751-3403362e5224\") " pod="openshift-multus/multus-additional-cni-plugins-7kkz9" Aug 05 16:21:27.429944 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428590 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/63118331-baa7-4178-9751-3403362e5224-os-release\") pod \"multus-additional-cni-plugins-7kkz9\" (UID: \"63118331-baa7-4178-9751-3403362e5224\") " pod="openshift-multus/multus-additional-cni-plugins-7kkz9" Aug 05 16:21:27.429944 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428609 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2a0163b-3977-4a52-8c23-6139f01a9b2d-host\") pod \"node-ca-bmvw9\" (UID: \"f2a0163b-3977-4a52-8c23-6139f01a9b2d\") " pod="openshift-image-registry/node-ca-bmvw9" Aug 05 16:21:27.429944 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428633 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-hostroot\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.429944 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428679 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-host-run-multus-certs\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.430913 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428695 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2a0163b-3977-4a52-8c23-6139f01a9b2d-host\") pod \"node-ca-bmvw9\" (UID: \"f2a0163b-3977-4a52-8c23-6139f01a9b2d\") " pod="openshift-image-registry/node-ca-bmvw9" Aug 05 16:21:27.430913 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428747 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-host-run-k8s-cni-cncf-io\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.430913 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428808 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-multus-conf-dir\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.430913 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428895 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5240cc7e-7437-476c-a63c-fd147054d87c-iptables-alerter-script\") pod \"iptables-alerter-8kt5n\" (UID: \"5240cc7e-7437-476c-a63c-fd147054d87c\") " pod="openshift-network-operator/iptables-alerter-8kt5n" Aug 05 16:21:27.430913 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428906 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f2a0163b-3977-4a52-8c23-6139f01a9b2d-serviceca\") pod \"node-ca-bmvw9\" (UID: \"f2a0163b-3977-4a52-8c23-6139f01a9b2d\") " pod="openshift-image-registry/node-ca-bmvw9" Aug 05 16:21:27.430913 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428937 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-multus-cni-dir\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.430913 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.428967 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/866c70f0-ff16-4ca9-b8db-730393bff7f4-os-release\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.430913 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.429041 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/93035306-7fa8-40fa-9517-2992e9340f56-etc-modprobe-d\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.430913 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:27.429043 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 16:21:27.430913 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:27.429101 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d233df-9262-45b2-8065-0d44a7ee9e12-original-pull-secret podName:39d233df-9262-45b2-8065-0d44a7ee9e12 nodeName:}" failed. No retries permitted until 2026-08-05 16:21:27.929084093 +0000 UTC m=+3.085929977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39d233df-9262-45b2-8065-0d44a7ee9e12-original-pull-secret") pod "global-pull-secret-syncer-qjdbg" (UID: "39d233df-9262-45b2-8065-0d44a7ee9e12") : object "kube-system"/"original-pull-secret" not registered Aug 05 16:21:27.430913 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.429294 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63118331-baa7-4178-9751-3403362e5224-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7kkz9\" (UID: \"63118331-baa7-4178-9751-3403362e5224\") " pod="openshift-multus/multus-additional-cni-plugins-7kkz9" Aug 05 16:21:27.430913 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.429307 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/866c70f0-ff16-4ca9-b8db-730393bff7f4-cni-binary-copy\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.430913 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.429549 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/63118331-baa7-4178-9751-3403362e5224-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7kkz9\" (UID: \"63118331-baa7-4178-9751-3403362e5224\") " pod="openshift-multus/multus-additional-cni-plugins-7kkz9" Aug 05 16:21:27.430913 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.429656 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63118331-baa7-4178-9751-3403362e5224-cni-binary-copy\") pod \"multus-additional-cni-plugins-7kkz9\" (UID: \"63118331-baa7-4178-9751-3403362e5224\") " pod="openshift-multus/multus-additional-cni-plugins-7kkz9" Aug 05 16:21:27.430913 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.429984 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/333a43c6-e0d3-42ab-a596-3670a69c8d5d-agent-certs\") pod \"konnectivity-agent-kpd4q\" (UID: \"333a43c6-e0d3-42ab-a596-3670a69c8d5d\") " pod="kube-system/konnectivity-agent-kpd4q" Aug 05 16:21:27.432335 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.432315 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93035306-7fa8-40fa-9517-2992e9340f56-tmp\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.432791 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.432702 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/93035306-7fa8-40fa-9517-2992e9340f56-etc-tuned\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.441317 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:27.440045 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 16:21:27.441317 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:27.440068 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 16:21:27.441317 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:27.440080 2573 projected.go:196] Error preparing data for projected volume kube-api-access-8x5w8 for pod openshift-network-diagnostics/network-check-target-pwx27: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:21:27.441317 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:27.440152 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e303e40d-9669-4230-a1e1-c75ea3a2c4df-kube-api-access-8x5w8 podName:e303e40d-9669-4230-a1e1-c75ea3a2c4df nodeName:}" failed. No retries permitted until 2026-08-05 16:21:27.940133682 +0000 UTC m=+3.096979570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8x5w8" (UniqueName: "kubernetes.io/projected/e303e40d-9669-4230-a1e1-c75ea3a2c4df-kube-api-access-8x5w8") pod "network-check-target-pwx27" (UID: "e303e40d-9669-4230-a1e1-c75ea3a2c4df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:21:27.441317 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.440861 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6bnc\" (UniqueName: \"kubernetes.io/projected/f2a0163b-3977-4a52-8c23-6139f01a9b2d-kube-api-access-z6bnc\") pod \"node-ca-bmvw9\" (UID: \"f2a0163b-3977-4a52-8c23-6139f01a9b2d\") " pod="openshift-image-registry/node-ca-bmvw9" Aug 05 16:21:27.446801 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.446757 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ts9s\" (UniqueName: \"kubernetes.io/projected/63118331-baa7-4178-9751-3403362e5224-kube-api-access-2ts9s\") pod \"multus-additional-cni-plugins-7kkz9\" (UID: \"63118331-baa7-4178-9751-3403362e5224\") " pod="openshift-multus/multus-additional-cni-plugins-7kkz9" Aug 05 16:21:27.448525 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.448445 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhvvk\" (UniqueName: \"kubernetes.io/projected/866c70f0-ff16-4ca9-b8db-730393bff7f4-kube-api-access-vhvvk\") pod \"multus-pfl76\" (UID: \"866c70f0-ff16-4ca9-b8db-730393bff7f4\") " pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.449155 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.449111 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62sl8\" (UniqueName: \"kubernetes.io/projected/aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d-kube-api-access-62sl8\") pod \"network-metrics-daemon-88qn5\" (UID: \"aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d\") " pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:27.449155 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.449120 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65zvj\" (UniqueName: \"kubernetes.io/projected/5240cc7e-7437-476c-a63c-fd147054d87c-kube-api-access-65zvj\") pod \"iptables-alerter-8kt5n\" (UID: \"5240cc7e-7437-476c-a63c-fd147054d87c\") " pod="openshift-network-operator/iptables-alerter-8kt5n" Aug 05 16:21:27.452927 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.452886 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvgjt\" (UniqueName: \"kubernetes.io/projected/93035306-7fa8-40fa-9517-2992e9340f56-kube-api-access-xvgjt\") pod \"tuned-xrv5x\" (UID: \"93035306-7fa8-40fa-9517-2992e9340f56\") " pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.522489 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.522455 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vbh4x" Aug 05 16:21:27.532689 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:27.532656 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe314431_1429_4319_b3a0_0c7287575cef.slice/crio-3908be2f6ee1aa9b89a9a6ef670870a42e35d5320fb93b804c9255bfef9f2a27 WatchSource:0}: Error finding container 3908be2f6ee1aa9b89a9a6ef670870a42e35d5320fb93b804c9255bfef9f2a27: Status 404 returned error can't find the container with id 3908be2f6ee1aa9b89a9a6ef670870a42e35d5320fb93b804c9255bfef9f2a27 Aug 05 16:21:27.534491 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.534469 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st" Aug 05 16:21:27.542583 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.542562 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:27.542817 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:27.542787 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod621aa7b0_79b4_4b5a_a8bf_447ebe0a2075.slice/crio-0b60fb15c88b071d99d6c32815b19cb9b4b30a63bacdfc02fe71232e74749f30 WatchSource:0}: Error finding container 0b60fb15c88b071d99d6c32815b19cb9b4b30a63bacdfc02fe71232e74749f30: Status 404 returned error can't find the container with id 0b60fb15c88b071d99d6c32815b19cb9b4b30a63bacdfc02fe71232e74749f30 Aug 05 16:21:27.550437 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.550410 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kpd4q" Aug 05 16:21:27.551147 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:27.551120 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod119a5576_d632_47e3_8692_4a4a0e1b7191.slice/crio-c35d32388c253d591ce3f807aef51bc9e924e0a56c9715c61560e46c92eaa2c0 WatchSource:0}: Error finding container c35d32388c253d591ce3f807aef51bc9e924e0a56c9715c61560e46c92eaa2c0: Status 404 returned error can't find the container with id c35d32388c253d591ce3f807aef51bc9e924e0a56c9715c61560e46c92eaa2c0 Aug 05 16:21:27.559425 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.559402 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pfl76" Aug 05 16:21:27.560229 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:27.560193 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod333a43c6_e0d3_42ab_a596_3670a69c8d5d.slice/crio-619b8a2746c04018585d780e2804e22789f70e47237d0288db4dcee4e4487030 WatchSource:0}: Error finding container 619b8a2746c04018585d780e2804e22789f70e47237d0288db4dcee4e4487030: Status 404 returned error can't find the container with id 619b8a2746c04018585d780e2804e22789f70e47237d0288db4dcee4e4487030 Aug 05 16:21:27.569263 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.569243 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" Aug 05 16:21:27.579169 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.579141 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8kt5n" Aug 05 16:21:27.588858 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.588834 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7kkz9" Aug 05 16:21:27.596546 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.596523 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bmvw9" Aug 05 16:21:27.931939 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.931858 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d233df-9262-45b2-8065-0d44a7ee9e12-original-pull-secret\") pod \"global-pull-secret-syncer-qjdbg\" (UID: \"39d233df-9262-45b2-8065-0d44a7ee9e12\") " pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:27.931939 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:27.931902 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d-metrics-certs\") pod \"network-metrics-daemon-88qn5\" (UID: \"aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d\") " pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:27.932139 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:27.932033 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 16:21:27.932139 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:27.932133 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d233df-9262-45b2-8065-0d44a7ee9e12-original-pull-secret podName:39d233df-9262-45b2-8065-0d44a7ee9e12 nodeName:}" failed. No retries permitted until 2026-08-05 16:21:28.932113716 +0000 UTC m=+4.088959620 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39d233df-9262-45b2-8065-0d44a7ee9e12-original-pull-secret") pod "global-pull-secret-syncer-qjdbg" (UID: "39d233df-9262-45b2-8065-0d44a7ee9e12") : object "kube-system"/"original-pull-secret" not registered Aug 05 16:21:27.932241 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:27.932039 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:21:27.932241 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:27.932206 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d-metrics-certs podName:aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d nodeName:}" failed. No retries permitted until 2026-08-05 16:21:28.9321897 +0000 UTC m=+4.089035580 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d-metrics-certs") pod "network-metrics-daemon-88qn5" (UID: "aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:21:28.032760 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:28.032700 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x5w8\" (UniqueName: \"kubernetes.io/projected/e303e40d-9669-4230-a1e1-c75ea3a2c4df-kube-api-access-8x5w8\") pod \"network-check-target-pwx27\" (UID: \"e303e40d-9669-4230-a1e1-c75ea3a2c4df\") " pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:21:28.032946 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:28.032853 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 16:21:28.032946 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:28.032879 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 16:21:28.032946 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:28.032895 2573 projected.go:196] Error preparing data for projected volume kube-api-access-8x5w8 for pod openshift-network-diagnostics/network-check-target-pwx27: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:21:28.033138 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:28.032960 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e303e40d-9669-4230-a1e1-c75ea3a2c4df-kube-api-access-8x5w8 podName:e303e40d-9669-4230-a1e1-c75ea3a2c4df nodeName:}" failed. No retries permitted until 2026-08-05 16:21:29.032941377 +0000 UTC m=+4.189787272 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8x5w8" (UniqueName: "kubernetes.io/projected/e303e40d-9669-4230-a1e1-c75ea3a2c4df-kube-api-access-8x5w8") pod "network-check-target-pwx27" (UID: "e303e40d-9669-4230-a1e1-c75ea3a2c4df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:21:28.137974 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:28.137939 2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 16:21:28.264686 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:28.264551 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-04 16:16:27 +0000 UTC" deadline="2028-02-19 21:36:43.123476248 +0000 UTC" Aug 05 16:21:28.264686 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:28.264633 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13517h15m14.858849615s" Aug 05 16:21:28.298594 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:28.298187 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:28.298594 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:28.298342 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88qn5" podUID="aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d" Aug 05 16:21:28.312326 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:28.312239 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kpd4q" event={"ID":"333a43c6-e0d3-42ab-a596-3670a69c8d5d","Type":"ContainerStarted","Data":"619b8a2746c04018585d780e2804e22789f70e47237d0288db4dcee4e4487030"} Aug 05 16:21:28.314120 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:28.314074 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" event={"ID":"119a5576-d632-47e3-8692-4a4a0e1b7191","Type":"ContainerStarted","Data":"c35d32388c253d591ce3f807aef51bc9e924e0a56c9715c61560e46c92eaa2c0"} Aug 05 16:21:28.315559 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:28.315509 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st" event={"ID":"621aa7b0-79b4-4b5a-a8bf-447ebe0a2075","Type":"ContainerStarted","Data":"0b60fb15c88b071d99d6c32815b19cb9b4b30a63bacdfc02fe71232e74749f30"} Aug 05 16:21:28.316967 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:28.316939 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vbh4x" event={"ID":"fe314431-1429-4319-b3a0-0c7287575cef","Type":"ContainerStarted","Data":"3908be2f6ee1aa9b89a9a6ef670870a42e35d5320fb93b804c9255bfef9f2a27"} Aug 05 16:21:28.941991 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:28.941958 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d233df-9262-45b2-8065-0d44a7ee9e12-original-pull-secret\") pod \"global-pull-secret-syncer-qjdbg\" (UID: \"39d233df-9262-45b2-8065-0d44a7ee9e12\") " pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:28.942118 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:28.942018 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d-metrics-certs\") pod \"network-metrics-daemon-88qn5\" (UID: \"aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d\") " pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:28.942181 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:28.942157 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:21:28.942230 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:28.942219 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d-metrics-certs podName:aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d nodeName:}" failed. No retries permitted until 2026-08-05 16:21:30.942200716 +0000 UTC m=+6.099046616 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d-metrics-certs") pod "network-metrics-daemon-88qn5" (UID: "aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:21:28.942639 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:28.942621 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 16:21:28.942724 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:28.942676 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d233df-9262-45b2-8065-0d44a7ee9e12-original-pull-secret podName:39d233df-9262-45b2-8065-0d44a7ee9e12 nodeName:}" failed. No retries permitted until 2026-08-05 16:21:30.942660683 +0000 UTC m=+6.099506567 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39d233df-9262-45b2-8065-0d44a7ee9e12-original-pull-secret") pod "global-pull-secret-syncer-qjdbg" (UID: "39d233df-9262-45b2-8065-0d44a7ee9e12") : object "kube-system"/"original-pull-secret" not registered Aug 05 16:21:29.042793 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:29.042757 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x5w8\" (UniqueName: \"kubernetes.io/projected/e303e40d-9669-4230-a1e1-c75ea3a2c4df-kube-api-access-8x5w8\") pod \"network-check-target-pwx27\" (UID: \"e303e40d-9669-4230-a1e1-c75ea3a2c4df\") " pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:21:29.043011 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:29.042989 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 16:21:29.043090 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:29.043020 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 16:21:29.043090 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:29.043033 2573 projected.go:196] Error preparing data for projected volume kube-api-access-8x5w8 for pod openshift-network-diagnostics/network-check-target-pwx27: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:21:29.043200 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:29.043097 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e303e40d-9669-4230-a1e1-c75ea3a2c4df-kube-api-access-8x5w8 podName:e303e40d-9669-4230-a1e1-c75ea3a2c4df nodeName:}" failed. No retries permitted until 2026-08-05 16:21:31.043076551 +0000 UTC m=+6.199922432 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8x5w8" (UniqueName: "kubernetes.io/projected/e303e40d-9669-4230-a1e1-c75ea3a2c4df-kube-api-access-8x5w8") pod "network-check-target-pwx27" (UID: "e303e40d-9669-4230-a1e1-c75ea3a2c4df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:21:29.047097 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:29.047064 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod866c70f0_ff16_4ca9_b8db_730393bff7f4.slice/crio-da3987026c4aba6eba0ec34fdb411d57790a9a35b37843fd459f466a1ab60a42 WatchSource:0}: Error finding container da3987026c4aba6eba0ec34fdb411d57790a9a35b37843fd459f466a1ab60a42: Status 404 returned error can't find the container with id da3987026c4aba6eba0ec34fdb411d57790a9a35b37843fd459f466a1ab60a42 Aug 05 16:21:29.052973 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:29.052947 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5240cc7e_7437_476c_a63c_fd147054d87c.slice/crio-02fcab48a6ff5b7e369ee798507822f5ce24760b5448e8d2a1f69ec09ce75559 WatchSource:0}: Error finding container 02fcab48a6ff5b7e369ee798507822f5ce24760b5448e8d2a1f69ec09ce75559: Status 404 returned error can't find the container with id 02fcab48a6ff5b7e369ee798507822f5ce24760b5448e8d2a1f69ec09ce75559 Aug 05 16:21:29.061199 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:29.061174 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93035306_7fa8_40fa_9517_2992e9340f56.slice/crio-44d8a2f5fe2ff2b93e2e6dd14db244a0ed9b32e98a8adb6859e9c711aa6f8648 WatchSource:0}: Error finding container 44d8a2f5fe2ff2b93e2e6dd14db244a0ed9b32e98a8adb6859e9c711aa6f8648: Status 404 returned error can't find the container with id 44d8a2f5fe2ff2b93e2e6dd14db244a0ed9b32e98a8adb6859e9c711aa6f8648 Aug 05 16:21:29.068064 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:29.067539 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63118331_baa7_4178_9751_3403362e5224.slice/crio-b2baca1ba9995d325bea633d514a82bab8df2929b582ffca69a3a837e95dcb01 WatchSource:0}: Error finding container b2baca1ba9995d325bea633d514a82bab8df2929b582ffca69a3a837e95dcb01: Status 404 returned error can't find the container with id b2baca1ba9995d325bea633d514a82bab8df2929b582ffca69a3a837e95dcb01 Aug 05 16:21:29.068064 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:21:29.067944 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a0163b_3977_4a52_8c23_6139f01a9b2d.slice/crio-ac935f3f9a3e1aeb1176ff7ccbaf53c94f609d5acf96176b4e88dde5d6e02290 WatchSource:0}: Error finding container ac935f3f9a3e1aeb1176ff7ccbaf53c94f609d5acf96176b4e88dde5d6e02290: Status 404 returned error can't find the container with id ac935f3f9a3e1aeb1176ff7ccbaf53c94f609d5acf96176b4e88dde5d6e02290 Aug 05 16:21:29.265203 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:29.264831 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-04 16:16:27 +0000 UTC" deadline="2028-03-21 08:09:36.176404423 +0000 UTC" Aug 05 16:21:29.265203 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:29.265086 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14247h48m6.911325506s" Aug 05 16:21:29.301404 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:29.300170 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:21:29.301404 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:29.300297 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwx27" podUID="e303e40d-9669-4230-a1e1-c75ea3a2c4df" Aug 05 16:21:29.301404 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:29.300740 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:29.301404 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:29.300843 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjdbg" podUID="39d233df-9262-45b2-8065-0d44a7ee9e12" Aug 05 16:21:29.324556 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:29.324528 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7kkz9" event={"ID":"63118331-baa7-4178-9751-3403362e5224","Type":"ContainerStarted","Data":"b2baca1ba9995d325bea633d514a82bab8df2929b582ffca69a3a837e95dcb01"} Aug 05 16:21:29.325813 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:29.325776 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bmvw9" event={"ID":"f2a0163b-3977-4a52-8c23-6139f01a9b2d","Type":"ContainerStarted","Data":"ac935f3f9a3e1aeb1176ff7ccbaf53c94f609d5acf96176b4e88dde5d6e02290"} Aug 05 16:21:29.328648 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:29.328590 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" event={"ID":"93035306-7fa8-40fa-9517-2992e9340f56","Type":"ContainerStarted","Data":"44d8a2f5fe2ff2b93e2e6dd14db244a0ed9b32e98a8adb6859e9c711aa6f8648"} Aug 05 16:21:29.330942 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:29.330778 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8kt5n" event={"ID":"5240cc7e-7437-476c-a63c-fd147054d87c","Type":"ContainerStarted","Data":"02fcab48a6ff5b7e369ee798507822f5ce24760b5448e8d2a1f69ec09ce75559"} Aug 05 16:21:29.333344 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:29.333282 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pfl76" event={"ID":"866c70f0-ff16-4ca9-b8db-730393bff7f4","Type":"ContainerStarted","Data":"da3987026c4aba6eba0ec34fdb411d57790a9a35b37843fd459f466a1ab60a42"} Aug 05 16:21:29.338698 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:29.337978 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-37.ec2.internal" event={"ID":"14d058665423739b2594851afe44a8fc","Type":"ContainerStarted","Data":"ff0c75b8074fb63e3b7f0a9f030999e8088330771ffd25c2ae11728bcbc42fc9"} Aug 05 16:21:30.298521 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:30.298487 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:30.299128 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:30.298633 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88qn5" podUID="aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d" Aug 05 16:21:30.342248 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:30.342196 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-37.ec2.internal" event={"ID":"4408f2cc25089e2ed0e44bb381e90c02","Type":"ContainerStarted","Data":"4f67d80a58603963a8c86bc19972168de97e899e246e66e97f23af038a9755c2"} Aug 05 16:21:30.369310 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:30.366822 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-37.ec2.internal" podStartSLOduration=4.36680391 podStartE2EDuration="4.36680391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:21:29.357909364 +0000 UTC m=+4.514755278" watchObservedRunningTime="2026-08-05 16:21:30.36680391 +0000 UTC m=+5.523649813" Aug 05 16:21:30.961645 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:30.961579 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d233df-9262-45b2-8065-0d44a7ee9e12-original-pull-secret\") pod \"global-pull-secret-syncer-qjdbg\" (UID: \"39d233df-9262-45b2-8065-0d44a7ee9e12\") " pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:30.961851 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:30.961693 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d-metrics-certs\") pod \"network-metrics-daemon-88qn5\" (UID: \"aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d\") " pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:30.961927 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:30.961862 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:21:30.961927 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:30.961923 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d-metrics-certs podName:aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d nodeName:}" failed. No retries permitted until 2026-08-05 16:21:34.961904331 +0000 UTC m=+10.118750212 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d-metrics-certs") pod "network-metrics-daemon-88qn5" (UID: "aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:21:30.962037 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:30.961929 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 16:21:30.962037 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:30.961992 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d233df-9262-45b2-8065-0d44a7ee9e12-original-pull-secret podName:39d233df-9262-45b2-8065-0d44a7ee9e12 nodeName:}" failed. No retries permitted until 2026-08-05 16:21:34.96197532 +0000 UTC m=+10.118821219 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39d233df-9262-45b2-8065-0d44a7ee9e12-original-pull-secret") pod "global-pull-secret-syncer-qjdbg" (UID: "39d233df-9262-45b2-8065-0d44a7ee9e12") : object "kube-system"/"original-pull-secret" not registered Aug 05 16:21:31.063097 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:31.063058 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x5w8\" (UniqueName: \"kubernetes.io/projected/e303e40d-9669-4230-a1e1-c75ea3a2c4df-kube-api-access-8x5w8\") pod \"network-check-target-pwx27\" (UID: \"e303e40d-9669-4230-a1e1-c75ea3a2c4df\") " pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:21:31.063285 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:31.063234 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 16:21:31.063285 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:31.063259 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 16:21:31.063285 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:31.063272 2573 projected.go:196] Error preparing data for projected volume kube-api-access-8x5w8 for pod openshift-network-diagnostics/network-check-target-pwx27: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:21:31.063484 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:31.063340 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e303e40d-9669-4230-a1e1-c75ea3a2c4df-kube-api-access-8x5w8 podName:e303e40d-9669-4230-a1e1-c75ea3a2c4df nodeName:}" failed. No retries permitted until 2026-08-05 16:21:35.063320853 +0000 UTC m=+10.220166755 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8x5w8" (UniqueName: "kubernetes.io/projected/e303e40d-9669-4230-a1e1-c75ea3a2c4df-kube-api-access-8x5w8") pod "network-check-target-pwx27" (UID: "e303e40d-9669-4230-a1e1-c75ea3a2c4df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:21:31.298991 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:31.298448 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:21:31.298991 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:31.298586 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwx27" podUID="e303e40d-9669-4230-a1e1-c75ea3a2c4df" Aug 05 16:21:31.298991 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:31.298634 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:31.298991 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:31.298738 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjdbg" podUID="39d233df-9262-45b2-8065-0d44a7ee9e12" Aug 05 16:21:32.298297 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:32.298260 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:32.298481 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:32.298434 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88qn5" podUID="aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d" Aug 05 16:21:32.348220 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:32.348183 2573 generic.go:356] "Generic (PLEG): container finished" podID="4408f2cc25089e2ed0e44bb381e90c02" containerID="4f67d80a58603963a8c86bc19972168de97e899e246e66e97f23af038a9755c2" exitCode=0 Aug 05 16:21:32.348683 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:32.348229 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-37.ec2.internal" event={"ID":"4408f2cc25089e2ed0e44bb381e90c02","Type":"ContainerDied","Data":"4f67d80a58603963a8c86bc19972168de97e899e246e66e97f23af038a9755c2"} Aug 05 16:21:33.299677 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:33.298589 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:33.299677 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:33.298722 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjdbg" podUID="39d233df-9262-45b2-8065-0d44a7ee9e12" Aug 05 16:21:33.299677 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:33.299205 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:21:33.299677 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:33.299303 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwx27" podUID="e303e40d-9669-4230-a1e1-c75ea3a2c4df" Aug 05 16:21:34.298481 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:34.298439 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:34.298956 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:34.298584 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88qn5" podUID="aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d" Aug 05 16:21:34.998303 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:34.997587 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d233df-9262-45b2-8065-0d44a7ee9e12-original-pull-secret\") pod \"global-pull-secret-syncer-qjdbg\" (UID: \"39d233df-9262-45b2-8065-0d44a7ee9e12\") " pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:34.998303 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:34.997664 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d-metrics-certs\") pod \"network-metrics-daemon-88qn5\" (UID: \"aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d\") " pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:34.998303 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:34.997795 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:21:34.998303 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:34.997855 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d-metrics-certs podName:aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d nodeName:}" failed. No retries permitted until 2026-08-05 16:21:42.997835499 +0000 UTC m=+18.154681393 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d-metrics-certs") pod "network-metrics-daemon-88qn5" (UID: "aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:21:34.998303 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:34.998222 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 16:21:34.998303 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:34.998267 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d233df-9262-45b2-8065-0d44a7ee9e12-original-pull-secret podName:39d233df-9262-45b2-8065-0d44a7ee9e12 nodeName:}" failed. No retries permitted until 2026-08-05 16:21:42.998253986 +0000 UTC m=+18.155099868 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39d233df-9262-45b2-8065-0d44a7ee9e12-original-pull-secret") pod "global-pull-secret-syncer-qjdbg" (UID: "39d233df-9262-45b2-8065-0d44a7ee9e12") : object "kube-system"/"original-pull-secret" not registered Aug 05 16:21:35.098331 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:35.098261 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x5w8\" (UniqueName: \"kubernetes.io/projected/e303e40d-9669-4230-a1e1-c75ea3a2c4df-kube-api-access-8x5w8\") pod \"network-check-target-pwx27\" (UID: \"e303e40d-9669-4230-a1e1-c75ea3a2c4df\") " pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:21:35.098552 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:35.098529 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 16:21:35.098552 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:35.098554 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 16:21:35.098702 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:35.098569 2573 projected.go:196] Error preparing data for projected volume kube-api-access-8x5w8 for pod openshift-network-diagnostics/network-check-target-pwx27: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:21:35.098702 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:35.098628 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e303e40d-9669-4230-a1e1-c75ea3a2c4df-kube-api-access-8x5w8 podName:e303e40d-9669-4230-a1e1-c75ea3a2c4df nodeName:}" failed. No retries permitted until 2026-08-05 16:21:43.098610248 +0000 UTC m=+18.255456144 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8x5w8" (UniqueName: "kubernetes.io/projected/e303e40d-9669-4230-a1e1-c75ea3a2c4df-kube-api-access-8x5w8") pod "network-check-target-pwx27" (UID: "e303e40d-9669-4230-a1e1-c75ea3a2c4df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:21:35.299109 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:35.299026 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:35.299697 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:35.299139 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjdbg" podUID="39d233df-9262-45b2-8065-0d44a7ee9e12" Aug 05 16:21:35.299697 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:35.299256 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:21:35.299697 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:35.299358 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwx27" podUID="e303e40d-9669-4230-a1e1-c75ea3a2c4df" Aug 05 16:21:35.357763 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:35.357721 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st" event={"ID":"621aa7b0-79b4-4b5a-a8bf-447ebe0a2075","Type":"ContainerStarted","Data":"4c622b49855c4cb7d4f970b06630f9a9376f40490b83143b6c7a2fbb094a2e7a"} Aug 05 16:21:35.360527 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:35.360497 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kpd4q" event={"ID":"333a43c6-e0d3-42ab-a596-3670a69c8d5d","Type":"ContainerStarted","Data":"0f33915b56dfff2e92b112a82677f432f5324fa19260e7964f9d277fe9d526a5"} Aug 05 16:21:35.362909 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:35.362884 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vbh4x" event={"ID":"fe314431-1429-4319-b3a0-0c7287575cef","Type":"ContainerStarted","Data":"310ec37f7b032b7fffbcee0b38b1defafb3e78d0f47f21c66c04a31cdaf91bc7"} Aug 05 16:21:35.365356 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:35.365330 2573 generic.go:356] "Generic (PLEG): container finished" podID="63118331-baa7-4178-9751-3403362e5224" containerID="c35ac894fb772682aaf4a419b2664ee6872f2e0cba3963d59dd21e7fc6cd1b4c" exitCode=0 Aug 05 16:21:35.365491 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:35.365425 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7kkz9" event={"ID":"63118331-baa7-4178-9751-3403362e5224","Type":"ContainerDied","Data":"c35ac894fb772682aaf4a419b2664ee6872f2e0cba3963d59dd21e7fc6cd1b4c"} Aug 05 16:21:35.368974 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:35.368753 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bmvw9" event={"ID":"f2a0163b-3977-4a52-8c23-6139f01a9b2d","Type":"ContainerStarted","Data":"61b2fe9dbf25e8073e1908bb47e88739f829ca047ca8e12752d6092418f99dd6"} Aug 05 16:21:35.371408 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:35.371386 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-37.ec2.internal" event={"ID":"4408f2cc25089e2ed0e44bb381e90c02","Type":"ContainerStarted","Data":"b7b9a2c9f633ed936185cdfe58929d790c1b8f21e0474c312d061b95ffff9a63"} Aug 05 16:21:35.379188 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:35.379129 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kpd4q" podStartSLOduration=3.349138323 podStartE2EDuration="10.379111909s" totalImagesPullingTime="7.029973586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:21:35.377230534 +0000 UTC m=+10.534076436" watchObservedRunningTime="2026-08-05 16:21:35.379111909 +0000 UTC m=+10.535957812" Aug 05 16:21:35.400367 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:35.400320 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vbh4x" podStartSLOduration=5.067850922 podStartE2EDuration="10.400304253s" totalImagesPullingTime="5.332453331s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:21:35.400071035 +0000 UTC m=+10.556916937" watchObservedRunningTime="2026-08-05 16:21:35.400304253 +0000 UTC m=+10.557150154" Aug 05 16:21:35.437958 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:35.437904 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bmvw9" podStartSLOduration=4.878125466 podStartE2EDuration="10.437883552s" totalImagesPullingTime="5.559758086s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:21:35.437024927 +0000 UTC m=+10.593870830" watchObservedRunningTime="2026-08-05 16:21:35.437883552 +0000 UTC m=+10.594729457" Aug 05 16:21:35.453850 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:35.453811 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-37.ec2.internal" podStartSLOduration=9.453798413 podStartE2EDuration="9.453798413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:21:35.453239867 +0000 UTC m=+10.610085768" watchObservedRunningTime="2026-08-05 16:21:35.453798413 +0000 UTC m=+10.610644314" Aug 05 16:21:36.298765 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:36.298734 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:36.298945 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:36.298843 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88qn5" podUID="aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d" Aug 05 16:21:36.376086 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:36.376048 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8kt5n" event={"ID":"5240cc7e-7437-476c-a63c-fd147054d87c","Type":"ContainerStarted","Data":"95ff87de0b20f52b0a50e79c3dd28e4c5988fdeb3d487d16e6c08e365a7c2452"} Aug 05 16:21:36.394451 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:36.394354 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8kt5n" podStartSLOduration=5.830638117 podStartE2EDuration="11.39433622s" totalImagesPullingTime="5.563698103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:21:36.393577505 +0000 UTC m=+11.550423406" watchObservedRunningTime="2026-08-05 16:21:36.39433622 +0000 UTC m=+11.551182121" Aug 05 16:21:37.298536 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:37.298190 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:37.298723 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:37.298557 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjdbg" podUID="39d233df-9262-45b2-8065-0d44a7ee9e12" Aug 05 16:21:37.298723 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:37.298219 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:21:37.298723 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:37.298642 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwx27" podUID="e303e40d-9669-4230-a1e1-c75ea3a2c4df" Aug 05 16:21:38.298744 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:38.298708 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:38.299126 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:38.298833 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88qn5" podUID="aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d" Aug 05 16:21:38.806991 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:38.806945 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kpd4q" Aug 05 16:21:38.807669 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:38.807645 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kpd4q" Aug 05 16:21:39.298621 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:39.298539 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:21:39.298825 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:39.298670 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwx27" podUID="e303e40d-9669-4230-a1e1-c75ea3a2c4df" Aug 05 16:21:39.298825 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:39.298721 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:39.299307 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:39.298833 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjdbg" podUID="39d233df-9262-45b2-8065-0d44a7ee9e12" Aug 05 16:21:40.298010 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:40.297980 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:40.298183 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:40.298099 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88qn5" podUID="aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d" Aug 05 16:21:41.298860 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:41.298832 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:41.299365 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:41.298838 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:21:41.299365 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:41.298927 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjdbg" podUID="39d233df-9262-45b2-8065-0d44a7ee9e12" Aug 05 16:21:41.299365 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:41.299014 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwx27" podUID="e303e40d-9669-4230-a1e1-c75ea3a2c4df" Aug 05 16:21:41.377048 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:41.377008 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kpd4q" Aug 05 16:21:41.377228 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:41.377134 2573 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Aug 05 16:21:41.377636 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:41.377611 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kpd4q" Aug 05 16:21:42.298752 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:42.298716 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:42.299002 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:42.298874 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88qn5" podUID="aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d" Aug 05 16:21:43.062542 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:43.062507 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d233df-9262-45b2-8065-0d44a7ee9e12-original-pull-secret\") pod \"global-pull-secret-syncer-qjdbg\" (UID: \"39d233df-9262-45b2-8065-0d44a7ee9e12\") " pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:43.062542 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:43.062557 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d-metrics-certs\") pod \"network-metrics-daemon-88qn5\" (UID: \"aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d\") " pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:43.062860 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:43.062668 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:21:43.062860 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:43.062734 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d-metrics-certs podName:aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d nodeName:}" failed. No retries permitted until 2026-08-05 16:21:59.062715807 +0000 UTC m=+34.219561689 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d-metrics-certs") pod "network-metrics-daemon-88qn5" (UID: "aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:21:43.062860 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:43.062669 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 16:21:43.062860 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:43.062820 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d233df-9262-45b2-8065-0d44a7ee9e12-original-pull-secret podName:39d233df-9262-45b2-8065-0d44a7ee9e12 nodeName:}" failed. No retries permitted until 2026-08-05 16:21:59.062802228 +0000 UTC m=+34.219648108 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39d233df-9262-45b2-8065-0d44a7ee9e12-original-pull-secret") pod "global-pull-secret-syncer-qjdbg" (UID: "39d233df-9262-45b2-8065-0d44a7ee9e12") : object "kube-system"/"original-pull-secret" not registered Aug 05 16:21:43.163412 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:43.163356 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x5w8\" (UniqueName: \"kubernetes.io/projected/e303e40d-9669-4230-a1e1-c75ea3a2c4df-kube-api-access-8x5w8\") pod \"network-check-target-pwx27\" (UID: \"e303e40d-9669-4230-a1e1-c75ea3a2c4df\") " pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:21:43.163596 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:43.163531 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 16:21:43.163596 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:43.163553 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 16:21:43.163596 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:43.163566 2573 projected.go:196] Error preparing data for projected volume kube-api-access-8x5w8 for pod openshift-network-diagnostics/network-check-target-pwx27: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:21:43.163751 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:43.163631 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e303e40d-9669-4230-a1e1-c75ea3a2c4df-kube-api-access-8x5w8 podName:e303e40d-9669-4230-a1e1-c75ea3a2c4df nodeName:}" failed. No retries permitted until 2026-08-05 16:21:59.163613752 +0000 UTC m=+34.320459654 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8x5w8" (UniqueName: "kubernetes.io/projected/e303e40d-9669-4230-a1e1-c75ea3a2c4df-kube-api-access-8x5w8") pod "network-check-target-pwx27" (UID: "e303e40d-9669-4230-a1e1-c75ea3a2c4df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:21:43.298450 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:43.298370 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:21:43.298450 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:43.298425 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:43.298667 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:43.298507 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwx27" podUID="e303e40d-9669-4230-a1e1-c75ea3a2c4df" Aug 05 16:21:43.298667 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:43.298617 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjdbg" podUID="39d233df-9262-45b2-8065-0d44a7ee9e12" Aug 05 16:21:44.298699 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:44.298668 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:44.299207 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:44.298795 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88qn5" podUID="aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d" Aug 05 16:21:45.298848 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:45.298823 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:45.299292 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:45.298916 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjdbg" podUID="39d233df-9262-45b2-8065-0d44a7ee9e12" Aug 05 16:21:45.299292 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:45.298993 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:21:45.299292 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:45.299074 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwx27" podUID="e303e40d-9669-4230-a1e1-c75ea3a2c4df" Aug 05 16:21:46.298642 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:46.298607 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:46.298833 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:46.298746 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88qn5" podUID="aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d" Aug 05 16:21:47.298093 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:47.298064 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:21:47.298549 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:47.298182 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwx27" podUID="e303e40d-9669-4230-a1e1-c75ea3a2c4df" Aug 05 16:21:47.298549 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:47.298540 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:47.298667 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:47.298636 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjdbg" podUID="39d233df-9262-45b2-8065-0d44a7ee9e12" Aug 05 16:21:48.298866 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:48.298835 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:48.299551 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:48.298967 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88qn5" podUID="aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d" Aug 05 16:21:48.617443 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:48.617284 2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 05 16:21:49.282573 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:49.282231 2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-05T16:21:48.617437773Z","UUID":"7dcd1e17-e48e-47f3-8f4a-e4cb9250bac8","Handler":null,"Name":"","Endpoint":""} Aug 05 16:21:49.283769 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:49.283746 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 05 16:21:49.283877 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:49.283793 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 05 16:21:49.297934 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:49.297913 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:21:49.298069 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:49.297948 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:49.298069 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:49.298044 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwx27" podUID="e303e40d-9669-4230-a1e1-c75ea3a2c4df" Aug 05 16:21:49.298184 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:49.298137 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjdbg" podUID="39d233df-9262-45b2-8065-0d44a7ee9e12" Aug 05 16:21:49.403104 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:49.403019 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pfl76" event={"ID":"866c70f0-ff16-4ca9-b8db-730393bff7f4","Type":"ContainerStarted","Data":"00b8c838bfb99643a49b41640c39d5f05cc56c4dbefc26fe66b29a44519a758e"} Aug 05 16:21:49.406427 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:49.406400 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" event={"ID":"119a5576-d632-47e3-8692-4a4a0e1b7191","Type":"ContainerStarted","Data":"b5146d74d560a3ce613b4e528df9018934141e6113004e1cdbec546b105b32dc"} Aug 05 16:21:49.406539 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:49.406476 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" event={"ID":"119a5576-d632-47e3-8692-4a4a0e1b7191","Type":"ContainerStarted","Data":"e8175fb7a25be311174c8aea77fd194dcc6106f484bd12039628e0755e487752"} Aug 05 16:21:49.406539 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:49.406526 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" event={"ID":"119a5576-d632-47e3-8692-4a4a0e1b7191","Type":"ContainerStarted","Data":"5a17d7a1a8cf698a9aa3ca5bd57f240527a77b18029baa97300c729182313a57"} Aug 05 16:21:49.406664 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:49.406573 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" event={"ID":"119a5576-d632-47e3-8692-4a4a0e1b7191","Type":"ContainerStarted","Data":"d4598bff47a147c921b273d8a5c50b1648a166289017d9b2e971d4bf84d4740c"} Aug 05 16:21:49.406664 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:49.406619 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" event={"ID":"119a5576-d632-47e3-8692-4a4a0e1b7191","Type":"ContainerStarted","Data":"14c06f065a2a6b55f11475abfa01bbd77485be942d339a693963c9b32752549b"} Aug 05 16:21:49.406756 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:49.406664 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" event={"ID":"119a5576-d632-47e3-8692-4a4a0e1b7191","Type":"ContainerStarted","Data":"9f523714f6488ce14a602dcf973d2c8cdbc300b0b7266bdf856efd49c749c29c"} Aug 05 16:21:49.408465 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:49.408441 2573 generic.go:356] "Generic (PLEG): container finished" podID="63118331-baa7-4178-9751-3403362e5224" containerID="147a30606fb12369064677ad569d514c2741beee729cd7b86ff2314c7c88f089" exitCode=0 Aug 05 16:21:49.408571 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:49.408510 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7kkz9" event={"ID":"63118331-baa7-4178-9751-3403362e5224","Type":"ContainerDied","Data":"147a30606fb12369064677ad569d514c2741beee729cd7b86ff2314c7c88f089"} Aug 05 16:21:49.410151 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:49.410125 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" event={"ID":"93035306-7fa8-40fa-9517-2992e9340f56","Type":"ContainerStarted","Data":"2976cd999188a46eb0984022753bacf327206130409a7173dddc27717321b5be"} Aug 05 16:21:49.412018 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:49.412000 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st" event={"ID":"621aa7b0-79b4-4b5a-a8bf-447ebe0a2075","Type":"ContainerStarted","Data":"c3cf47ab8abfee52411db5830134f749ee55388ed3f5df26ddef46fbc78d27de"} Aug 05 16:21:49.423894 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:49.423851 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pfl76" podStartSLOduration=4.951704792 podStartE2EDuration="24.42383834s" totalImagesPullingTime="19.472133548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:21:49.423338097 +0000 UTC m=+24.580183999" watchObservedRunningTime="2026-08-05 16:21:49.42383834 +0000 UTC m=+24.580684241" Aug 05 16:21:49.463670 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:49.463620 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xrv5x" podStartSLOduration=4.676646696 podStartE2EDuration="24.463608771s" totalImagesPullingTime="19.786962075s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:21:49.463489431 +0000 UTC m=+24.620335345" watchObservedRunningTime="2026-08-05 16:21:49.463608771 +0000 UTC m=+24.620454672" Aug 05 16:21:50.298472 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:50.298405 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:50.298624 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:50.298511 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88qn5" podUID="aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d" Aug 05 16:21:50.416591 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:50.416551 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st" event={"ID":"621aa7b0-79b4-4b5a-a8bf-447ebe0a2075","Type":"ContainerStarted","Data":"9519c9ed8b6022b120255ae9d3012ad181d93dc86fb61aa3f03cee13ca12c34f"} Aug 05 16:21:50.438966 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:50.438925 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bb4st" podStartSLOduration=3.445032519 podStartE2EDuration="25.438910804s" totalImagesPullingTime="21.993878285s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:21:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:21:50.438614109 +0000 UTC m=+25.595460011" watchObservedRunningTime="2026-08-05 16:21:50.438910804 +0000 UTC m=+25.595756722" Aug 05 16:21:51.298722 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:51.298700 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:21:51.298826 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:51.298700 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:51.298826 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:51.298807 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwx27" podUID="e303e40d-9669-4230-a1e1-c75ea3a2c4df" Aug 05 16:21:51.298903 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:51.298855 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjdbg" podUID="39d233df-9262-45b2-8065-0d44a7ee9e12" Aug 05 16:21:51.422237 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:51.422205 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" event={"ID":"119a5576-d632-47e3-8692-4a4a0e1b7191","Type":"ContainerStarted","Data":"676c0afcd7fefd79f18f20e9e3e95ad01cddc3d5b315bdb13e15c8c259f6d1f9"} Aug 05 16:21:51.424318 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:51.424293 2573 generic.go:356] "Generic (PLEG): container finished" podID="63118331-baa7-4178-9751-3403362e5224" containerID="5cf6b9be2470bcd8a8718e0fab19be6aad85055257b55f391b42dda9776d5fce" exitCode=0 Aug 05 16:21:51.424441 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:51.424347 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7kkz9" event={"ID":"63118331-baa7-4178-9751-3403362e5224","Type":"ContainerDied","Data":"5cf6b9be2470bcd8a8718e0fab19be6aad85055257b55f391b42dda9776d5fce"} Aug 05 16:21:52.298861 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:52.298829 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:52.299018 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:52.298935 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88qn5" podUID="aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d" Aug 05 16:21:53.298484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:53.298450 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:53.298883 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:53.298582 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjdbg" podUID="39d233df-9262-45b2-8065-0d44a7ee9e12" Aug 05 16:21:53.298883 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:53.298634 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:21:53.298883 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:53.298715 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwx27" podUID="e303e40d-9669-4230-a1e1-c75ea3a2c4df" Aug 05 16:21:53.431210 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:53.431178 2573 generic.go:356] "Generic (PLEG): container finished" podID="63118331-baa7-4178-9751-3403362e5224" containerID="4d05953c4b83eb51e234014667dd353fb7417f9c1aa4155d0d1e2e4dc7c28f59" exitCode=0 Aug 05 16:21:53.431355 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:53.431218 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7kkz9" event={"ID":"63118331-baa7-4178-9751-3403362e5224","Type":"ContainerDied","Data":"4d05953c4b83eb51e234014667dd353fb7417f9c1aa4155d0d1e2e4dc7c28f59"} Aug 05 16:21:54.298175 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:54.298001 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:54.298350 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:54.298249 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88qn5" podUID="aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d" Aug 05 16:21:54.437943 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:54.437892 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" event={"ID":"119a5576-d632-47e3-8692-4a4a0e1b7191","Type":"ContainerStarted","Data":"e94dfaf296dd90d95e396268a42926b39a2f72d8d954ad0c9a54406a66147383"} Aug 05 16:21:54.438493 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:54.438155 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:54.456610 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:54.456583 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:54.471574 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:54.471524 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" podStartSLOduration=8.526406623 podStartE2EDuration="29.471507649s" totalImagesPullingTime="20.945101026s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:21:54.471268935 +0000 UTC m=+29.628114838" watchObservedRunningTime="2026-08-05 16:21:54.471507649 +0000 UTC m=+29.628353551" Aug 05 16:21:55.300647 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:55.299957 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:55.300647 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:55.300103 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjdbg" podUID="39d233df-9262-45b2-8065-0d44a7ee9e12" Aug 05 16:21:55.300647 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:55.300526 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:21:55.300647 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:55.300606 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwx27" podUID="e303e40d-9669-4230-a1e1-c75ea3a2c4df" Aug 05 16:21:55.440393 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:55.440348 2573 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Aug 05 16:21:55.440918 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:55.440894 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:55.457322 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:55.457294 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:55.786886 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:55.786692 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qjdbg"] Aug 05 16:21:55.787867 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:55.787197 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:55.787867 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:55.787319 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjdbg" podUID="39d233df-9262-45b2-8065-0d44a7ee9e12" Aug 05 16:21:55.787867 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:55.787671 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pwx27"] Aug 05 16:21:55.787867 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:55.787740 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:21:55.787867 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:55.787824 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwx27" podUID="e303e40d-9669-4230-a1e1-c75ea3a2c4df" Aug 05 16:21:55.788202 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:55.788156 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-88qn5"] Aug 05 16:21:55.788270 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:55.788254 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:55.788431 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:55.788360 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88qn5" podUID="aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d" Aug 05 16:21:56.443472 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:56.443442 2573 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Aug 05 16:21:56.976798 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:56.976760 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:21:57.298893 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:57.298861 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:21:57.299073 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:57.298907 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:57.299073 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:57.298861 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:57.299073 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:57.298994 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwx27" podUID="e303e40d-9669-4230-a1e1-c75ea3a2c4df" Aug 05 16:21:57.299214 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:57.299088 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88qn5" podUID="aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d" Aug 05 16:21:57.299214 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:57.299184 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjdbg" podUID="39d233df-9262-45b2-8065-0d44a7ee9e12" Aug 05 16:21:57.356245 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:57.356213 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:21:57.449135 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:57.449089 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:21:57.549612 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:57.549536 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:21:58.255138 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:58.254968 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:21:58.450431 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:58.450336 2573 generic.go:356] "Generic (PLEG): container finished" podID="63118331-baa7-4178-9751-3403362e5224" containerID="bc9a142ec026d353cf01846a44f3a07fb3a5c9f335b8b830eed1da3bce691074" exitCode=0 Aug 05 16:21:58.450431 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:58.450408 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7kkz9" event={"ID":"63118331-baa7-4178-9751-3403362e5224","Type":"ContainerDied","Data":"bc9a142ec026d353cf01846a44f3a07fb3a5c9f335b8b830eed1da3bce691074"} Aug 05 16:21:58.460988 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:58.460958 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" podUID="119a5576-d632-47e3-8692-4a4a0e1b7191" containerName="ovnkube-controller" probeResult="failure" output="" Aug 05 16:21:59.083103 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:59.083074 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d233df-9262-45b2-8065-0d44a7ee9e12-original-pull-secret\") pod \"global-pull-secret-syncer-qjdbg\" (UID: \"39d233df-9262-45b2-8065-0d44a7ee9e12\") " pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:59.083300 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:59.083115 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d-metrics-certs\") pod \"network-metrics-daemon-88qn5\" (UID: \"aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d\") " pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:59.083300 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:59.083241 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 16:21:59.083300 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:59.083275 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:21:59.083483 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:59.083310 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d233df-9262-45b2-8065-0d44a7ee9e12-original-pull-secret podName:39d233df-9262-45b2-8065-0d44a7ee9e12 nodeName:}" failed. No retries permitted until 2026-08-05 16:22:31.083289752 +0000 UTC m=+66.240135645 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39d233df-9262-45b2-8065-0d44a7ee9e12-original-pull-secret") pod "global-pull-secret-syncer-qjdbg" (UID: "39d233df-9262-45b2-8065-0d44a7ee9e12") : object "kube-system"/"original-pull-secret" not registered Aug 05 16:21:59.083483 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:59.083328 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d-metrics-certs podName:aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d nodeName:}" failed. No retries permitted until 2026-08-05 16:22:31.083319033 +0000 UTC m=+66.240164917 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d-metrics-certs") pod "network-metrics-daemon-88qn5" (UID: "aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 16:21:59.183970 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:59.183942 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x5w8\" (UniqueName: \"kubernetes.io/projected/e303e40d-9669-4230-a1e1-c75ea3a2c4df-kube-api-access-8x5w8\") pod \"network-check-target-pwx27\" (UID: \"e303e40d-9669-4230-a1e1-c75ea3a2c4df\") " pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:21:59.184149 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:59.184124 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 16:21:59.184149 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:59.184147 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 16:21:59.184257 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:59.184160 2573 projected.go:196] Error preparing data for projected volume kube-api-access-8x5w8 for pod openshift-network-diagnostics/network-check-target-pwx27: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:21:59.184257 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:59.184221 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e303e40d-9669-4230-a1e1-c75ea3a2c4df-kube-api-access-8x5w8 podName:e303e40d-9669-4230-a1e1-c75ea3a2c4df nodeName:}" failed. No retries permitted until 2026-08-05 16:22:31.184203039 +0000 UTC m=+66.341048934 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8x5w8" (UniqueName: "kubernetes.io/projected/e303e40d-9669-4230-a1e1-c75ea3a2c4df-kube-api-access-8x5w8") pod "network-check-target-pwx27" (UID: "e303e40d-9669-4230-a1e1-c75ea3a2c4df") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 16:21:59.298179 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:59.298148 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:21:59.298356 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:59.298148 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:21:59.298356 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:59.298262 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:21:59.298356 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:59.298269 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88qn5" podUID="aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d" Aug 05 16:21:59.298356 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:59.298329 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwx27" podUID="e303e40d-9669-4230-a1e1-c75ea3a2c4df" Aug 05 16:21:59.298578 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:21:59.298405 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjdbg" podUID="39d233df-9262-45b2-8065-0d44a7ee9e12" Aug 05 16:21:59.455317 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:59.455287 2573 generic.go:356] "Generic (PLEG): container finished" podID="63118331-baa7-4178-9751-3403362e5224" containerID="d2bf3ea5028e4cf1b69ca6cae894e2434070f808a564e046263fd0465b2862ec" exitCode=0 Aug 05 16:21:59.455830 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:21:59.455341 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7kkz9" event={"ID":"63118331-baa7-4178-9751-3403362e5224","Type":"ContainerDied","Data":"d2bf3ea5028e4cf1b69ca6cae894e2434070f808a564e046263fd0465b2862ec"} Aug 05 16:22:00.460783 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:00.460746 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7kkz9" event={"ID":"63118331-baa7-4178-9751-3403362e5224","Type":"ContainerStarted","Data":"a6b08de719cc39a18ee9b5979a0d16d4198f69d13001f24d6a5cffe8fc1765d1"} Aug 05 16:22:00.486032 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:00.485981 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7kkz9" podStartSLOduration=9.72969396 podStartE2EDuration="35.485966751s" totalImagesPullingTime="25.756272791s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:21:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:22:00.484495171 +0000 UTC m=+35.641341071" watchObservedRunningTime="2026-08-05 16:22:00.485966751 +0000 UTC m=+35.642812649" Aug 05 16:22:00.608753 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:00.608721 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:00.668232 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:00.668185 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:00.798551 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:00.798525 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:00.914603 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:00.914567 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:01.012126 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.012099 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:01.200540 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.200464 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-37.ec2.internal" event="NodeReady" Aug 05 16:22:01.200704 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.200595 2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 05 16:22:01.255279 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.255248 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w8666"] Aug 05 16:22:01.280342 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.280313 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-477z7"] Aug 05 16:22:01.280481 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.280434 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w8666" Aug 05 16:22:01.283613 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.283592 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cnw72\"" Aug 05 16:22:01.284440 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.284426 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 05 16:22:01.284490 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.284441 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 05 16:22:01.294265 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.294246 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-477z7"] Aug 05 16:22:01.294367 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.294274 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w8666"] Aug 05 16:22:01.294367 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.294353 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-477z7" Aug 05 16:22:01.297967 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.297950 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:22:01.297967 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.297964 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:22:01.298153 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.297970 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:22:01.298260 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.298238 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 05 16:22:01.298347 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.298302 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5c2w7\"" Aug 05 16:22:01.299004 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.298481 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 05 16:22:01.299004 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.298555 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 05 16:22:01.301055 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.301040 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 05 16:22:01.301786 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.301642 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w287q\"" Aug 05 16:22:01.301786 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.301661 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 05 16:22:01.301786 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.301672 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 05 16:22:01.301786 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.301689 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 05 16:22:01.301786 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.301689 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-64w4b\"" Aug 05 16:22:01.398416 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.398353 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-metrics-tls\") pod \"dns-default-w8666\" (UID: \"89e8400b-bfeb-44c4-8302-59ab47d1b4cf\") " pod="openshift-dns/dns-default-w8666" Aug 05 16:22:01.398629 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.398500 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g456k\" (UniqueName: \"kubernetes.io/projected/76352a33-7d75-494f-9741-931a2cc97c40-kube-api-access-g456k\") pod \"ingress-canary-477z7\" (UID: \"76352a33-7d75-494f-9741-931a2cc97c40\") " pod="openshift-ingress-canary/ingress-canary-477z7" Aug 05 16:22:01.398629 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.398538 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-tmp-dir\") pod \"dns-default-w8666\" (UID: \"89e8400b-bfeb-44c4-8302-59ab47d1b4cf\") " pod="openshift-dns/dns-default-w8666" Aug 05 16:22:01.398629 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.398563 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6t7x\" (UniqueName: \"kubernetes.io/projected/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-kube-api-access-c6t7x\") pod \"dns-default-w8666\" (UID: \"89e8400b-bfeb-44c4-8302-59ab47d1b4cf\") " pod="openshift-dns/dns-default-w8666" Aug 05 16:22:01.398629 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.398607 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/76352a33-7d75-494f-9741-931a2cc97c40-cert\") pod \"ingress-canary-477z7\" (UID: \"76352a33-7d75-494f-9741-931a2cc97c40\") " pod="openshift-ingress-canary/ingress-canary-477z7" Aug 05 16:22:01.398779 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.398644 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-config-volume\") pod \"dns-default-w8666\" (UID: \"89e8400b-bfeb-44c4-8302-59ab47d1b4cf\") " pod="openshift-dns/dns-default-w8666" Aug 05 16:22:01.499667 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.499591 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g456k\" (UniqueName: \"kubernetes.io/projected/76352a33-7d75-494f-9741-931a2cc97c40-kube-api-access-g456k\") pod \"ingress-canary-477z7\" (UID: \"76352a33-7d75-494f-9741-931a2cc97c40\") " pod="openshift-ingress-canary/ingress-canary-477z7" Aug 05 16:22:01.499667 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.499627 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-tmp-dir\") pod \"dns-default-w8666\" (UID: \"89e8400b-bfeb-44c4-8302-59ab47d1b4cf\") " pod="openshift-dns/dns-default-w8666" Aug 05 16:22:01.499667 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.499645 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6t7x\" (UniqueName: \"kubernetes.io/projected/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-kube-api-access-c6t7x\") pod \"dns-default-w8666\" (UID: \"89e8400b-bfeb-44c4-8302-59ab47d1b4cf\") " pod="openshift-dns/dns-default-w8666" Aug 05 16:22:01.500132 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.499674 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/76352a33-7d75-494f-9741-931a2cc97c40-cert\") pod \"ingress-canary-477z7\" (UID: \"76352a33-7d75-494f-9741-931a2cc97c40\") " pod="openshift-ingress-canary/ingress-canary-477z7" Aug 05 16:22:01.500132 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.499811 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-config-volume\") pod \"dns-default-w8666\" (UID: \"89e8400b-bfeb-44c4-8302-59ab47d1b4cf\") " pod="openshift-dns/dns-default-w8666" Aug 05 16:22:01.500132 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.499875 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-metrics-tls\") pod \"dns-default-w8666\" (UID: \"89e8400b-bfeb-44c4-8302-59ab47d1b4cf\") " pod="openshift-dns/dns-default-w8666" Aug 05 16:22:01.500132 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.499988 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-tmp-dir\") pod \"dns-default-w8666\" (UID: \"89e8400b-bfeb-44c4-8302-59ab47d1b4cf\") " pod="openshift-dns/dns-default-w8666" Aug 05 16:22:01.500132 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:01.499993 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 16:22:01.500132 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:01.500060 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-metrics-tls podName:89e8400b-bfeb-44c4-8302-59ab47d1b4cf nodeName:}" failed. No retries permitted until 2026-08-05 16:22:02.000046469 +0000 UTC m=+37.156892351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-metrics-tls") pod "dns-default-w8666" (UID: "89e8400b-bfeb-44c4-8302-59ab47d1b4cf") : secret "dns-default-metrics-tls" not found Aug 05 16:22:01.500553 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.500532 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-config-volume\") pod \"dns-default-w8666\" (UID: \"89e8400b-bfeb-44c4-8302-59ab47d1b4cf\") " pod="openshift-dns/dns-default-w8666" Aug 05 16:22:01.503644 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.503628 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/76352a33-7d75-494f-9741-931a2cc97c40-cert\") pod \"ingress-canary-477z7\" (UID: \"76352a33-7d75-494f-9741-931a2cc97c40\") " pod="openshift-ingress-canary/ingress-canary-477z7" Aug 05 16:22:01.509752 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.509730 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6t7x\" (UniqueName: \"kubernetes.io/projected/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-kube-api-access-c6t7x\") pod \"dns-default-w8666\" (UID: \"89e8400b-bfeb-44c4-8302-59ab47d1b4cf\") " pod="openshift-dns/dns-default-w8666" Aug 05 16:22:01.509854 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.509835 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g456k\" (UniqueName: \"kubernetes.io/projected/76352a33-7d75-494f-9741-931a2cc97c40-kube-api-access-g456k\") pod \"ingress-canary-477z7\" (UID: \"76352a33-7d75-494f-9741-931a2cc97c40\") " pod="openshift-ingress-canary/ingress-canary-477z7" Aug 05 16:22:01.604589 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.604557 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-477z7" Aug 05 16:22:01.788594 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:01.788427 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-477z7"] Aug 05 16:22:01.792138 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:22:01.792114 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76352a33_7d75_494f_9741_931a2cc97c40.slice/crio-4d12637f5d6ffd074d0aa6246ef5e6a4eaca5d7c75708a34606a4a016e6a0086 WatchSource:0}: Error finding container 4d12637f5d6ffd074d0aa6246ef5e6a4eaca5d7c75708a34606a4a016e6a0086: Status 404 returned error can't find the container with id 4d12637f5d6ffd074d0aa6246ef5e6a4eaca5d7c75708a34606a4a016e6a0086 Aug 05 16:22:02.003474 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:02.003435 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-metrics-tls\") pod \"dns-default-w8666\" (UID: \"89e8400b-bfeb-44c4-8302-59ab47d1b4cf\") " pod="openshift-dns/dns-default-w8666" Aug 05 16:22:02.003694 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:02.003573 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 16:22:02.003694 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:02.003656 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-metrics-tls podName:89e8400b-bfeb-44c4-8302-59ab47d1b4cf nodeName:}" failed. No retries permitted until 2026-08-05 16:22:03.003632643 +0000 UTC m=+38.160478537 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-metrics-tls") pod "dns-default-w8666" (UID: "89e8400b-bfeb-44c4-8302-59ab47d1b4cf") : secret "dns-default-metrics-tls" not found Aug 05 16:22:02.466145 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:02.466113 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-477z7" event={"ID":"76352a33-7d75-494f-9741-931a2cc97c40","Type":"ContainerStarted","Data":"4d12637f5d6ffd074d0aa6246ef5e6a4eaca5d7c75708a34606a4a016e6a0086"} Aug 05 16:22:03.011491 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:03.011453 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-metrics-tls\") pod \"dns-default-w8666\" (UID: \"89e8400b-bfeb-44c4-8302-59ab47d1b4cf\") " pod="openshift-dns/dns-default-w8666" Aug 05 16:22:03.011995 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:03.011581 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 16:22:03.011995 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:03.011633 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-metrics-tls podName:89e8400b-bfeb-44c4-8302-59ab47d1b4cf nodeName:}" failed. No retries permitted until 2026-08-05 16:22:05.011620903 +0000 UTC m=+40.168466782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-metrics-tls") pod "dns-default-w8666" (UID: "89e8400b-bfeb-44c4-8302-59ab47d1b4cf") : secret "dns-default-metrics-tls" not found Aug 05 16:22:05.026927 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.026754 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-metrics-tls\") pod \"dns-default-w8666\" (UID: \"89e8400b-bfeb-44c4-8302-59ab47d1b4cf\") " pod="openshift-dns/dns-default-w8666" Aug 05 16:22:05.027290 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:05.026899 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 16:22:05.027290 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:05.026994 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-metrics-tls podName:89e8400b-bfeb-44c4-8302-59ab47d1b4cf nodeName:}" failed. No retries permitted until 2026-08-05 16:22:09.026978562 +0000 UTC m=+44.183824441 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-metrics-tls") pod "dns-default-w8666" (UID: "89e8400b-bfeb-44c4-8302-59ab47d1b4cf") : secret "dns-default-metrics-tls" not found Aug 05 16:22:05.392348 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.392318 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:05.474451 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.474356 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-477z7" event={"ID":"76352a33-7d75-494f-9741-931a2cc97c40","Type":"ContainerStarted","Data":"bd5d91f11469e57e583ce5fd20d726c3f4009406b80918b72c6e2fbcddda87f8"} Aug 05 16:22:05.486796 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.486772 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-8vzp6"] Aug 05 16:22:05.498080 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.498054 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:05.502489 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.502414 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-8vzp6"] Aug 05 16:22:05.502607 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.502494 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-8vzp6" Aug 05 16:22:05.503029 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.502983 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-477z7" podStartSLOduration=1.5046349509999999 podStartE2EDuration="4.502966585s" totalImagesPullingTime="2.998331634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:22:05.502827093 +0000 UTC m=+40.659672994" watchObservedRunningTime="2026-08-05 16:22:05.502966585 +0000 UTC m=+40.659812489" Aug 05 16:22:05.505631 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.505611 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-hn5w6\"" Aug 05 16:22:05.505718 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.505702 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Aug 05 16:22:05.507116 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.507098 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Aug 05 16:22:05.578044 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.578014 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-stpfk"] Aug 05 16:22:05.602461 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.602432 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cjmqg"] Aug 05 16:22:05.602602 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.602554 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-stpfk" Aug 05 16:22:05.608992 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.608927 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Aug 05 16:22:05.609274 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.609254 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-xx7t8\"" Aug 05 16:22:05.609883 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.609867 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Aug 05 16:22:05.609975 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.609886 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Aug 05 16:22:05.610093 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.610082 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Aug 05 16:22:05.626445 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.626422 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-stpfk"] Aug 05 16:22:05.626556 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.626452 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cjmqg"] Aug 05 16:22:05.626556 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.626539 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cjmqg" Aug 05 16:22:05.630494 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.630476 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Aug 05 16:22:05.630579 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.630546 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Aug 05 16:22:05.630848 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.630821 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-jq2dl\"" Aug 05 16:22:05.630968 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.630949 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs82j\" (UniqueName: \"kubernetes.io/projected/0e863683-b5a0-47e6-9d12-4a8539c943f7-kube-api-access-zs82j\") pod \"volume-data-source-validator-5b8776669-8vzp6\" (UID: \"0e863683-b5a0-47e6-9d12-4a8539c943f7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-8vzp6" Aug 05 16:22:05.631060 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.631043 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Aug 05 16:22:05.732585 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.732498 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/791538ee-cf4d-4198-aed7-9ac30d92460e-config\") pod \"service-ca-operator-784f97bcf6-stpfk\" (UID: \"791538ee-cf4d-4198-aed7-9ac30d92460e\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-stpfk" Aug 05 16:22:05.732585 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.732529 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb6wq\" (UniqueName: \"kubernetes.io/projected/791538ee-cf4d-4198-aed7-9ac30d92460e-kube-api-access-rb6wq\") pod \"service-ca-operator-784f97bcf6-stpfk\" (UID: \"791538ee-cf4d-4198-aed7-9ac30d92460e\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-stpfk" Aug 05 16:22:05.732585 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.732554 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdnv4\" (UniqueName: \"kubernetes.io/projected/0dd9c938-5048-43f1-ab25-4898652787c9-kube-api-access-cdnv4\") pod \"cluster-samples-operator-559cbf5745-cjmqg\" (UID: \"0dd9c938-5048-43f1-ab25-4898652787c9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cjmqg" Aug 05 16:22:05.732852 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.732664 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs82j\" (UniqueName: \"kubernetes.io/projected/0e863683-b5a0-47e6-9d12-4a8539c943f7-kube-api-access-zs82j\") pod \"volume-data-source-validator-5b8776669-8vzp6\" (UID: \"0e863683-b5a0-47e6-9d12-4a8539c943f7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-8vzp6" Aug 05 16:22:05.732852 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.732701 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/791538ee-cf4d-4198-aed7-9ac30d92460e-serving-cert\") pod \"service-ca-operator-784f97bcf6-stpfk\" (UID: \"791538ee-cf4d-4198-aed7-9ac30d92460e\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-stpfk" Aug 05 16:22:05.732852 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.732735 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dd9c938-5048-43f1-ab25-4898652787c9-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-cjmqg\" (UID: \"0dd9c938-5048-43f1-ab25-4898652787c9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cjmqg" Aug 05 16:22:05.751644 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.751619 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs82j\" (UniqueName: \"kubernetes.io/projected/0e863683-b5a0-47e6-9d12-4a8539c943f7-kube-api-access-zs82j\") pod \"volume-data-source-validator-5b8776669-8vzp6\" (UID: \"0e863683-b5a0-47e6-9d12-4a8539c943f7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-8vzp6" Aug 05 16:22:05.773191 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.773163 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-cbf86bfcb-8pv7r"] Aug 05 16:22:05.791507 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.791476 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:05.794484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.794464 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-cbf86bfcb-8pv7r"] Aug 05 16:22:05.794590 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.794547 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:05.798250 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.798229 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Aug 05 16:22:05.798352 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.798229 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Aug 05 16:22:05.798763 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.798749 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 05 16:22:05.799669 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.799655 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Aug 05 16:22:05.799746 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.799689 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-hdgcl\"" Aug 05 16:22:05.799815 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.799691 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Aug 05 16:22:05.799968 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.799952 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 05 16:22:05.814351 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.814333 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-8vzp6" Aug 05 16:22:05.833309 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.833284 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk72j\" (UniqueName: \"kubernetes.io/projected/773c2ff3-528d-4d42-a88c-03331f44791c-kube-api-access-zk72j\") pod \"router-default-cbf86bfcb-8pv7r\" (UID: \"773c2ff3-528d-4d42-a88c-03331f44791c\") " pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:05.833422 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.833338 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/791538ee-cf4d-4198-aed7-9ac30d92460e-serving-cert\") pod \"service-ca-operator-784f97bcf6-stpfk\" (UID: \"791538ee-cf4d-4198-aed7-9ac30d92460e\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-stpfk" Aug 05 16:22:05.833422 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.833355 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/773c2ff3-528d-4d42-a88c-03331f44791c-stats-auth\") pod \"router-default-cbf86bfcb-8pv7r\" (UID: \"773c2ff3-528d-4d42-a88c-03331f44791c\") " pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:05.833555 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.833418 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dd9c938-5048-43f1-ab25-4898652787c9-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-cjmqg\" (UID: \"0dd9c938-5048-43f1-ab25-4898652787c9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cjmqg" Aug 05 16:22:05.833555 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.833493 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/773c2ff3-528d-4d42-a88c-03331f44791c-default-certificate\") pod \"router-default-cbf86bfcb-8pv7r\" (UID: \"773c2ff3-528d-4d42-a88c-03331f44791c\") " pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:05.833555 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:05.833534 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 05 16:22:05.833702 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.833532 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/791538ee-cf4d-4198-aed7-9ac30d92460e-config\") pod \"service-ca-operator-784f97bcf6-stpfk\" (UID: \"791538ee-cf4d-4198-aed7-9ac30d92460e\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-stpfk" Aug 05 16:22:05.833702 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:05.833586 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dd9c938-5048-43f1-ab25-4898652787c9-samples-operator-tls podName:0dd9c938-5048-43f1-ab25-4898652787c9 nodeName:}" failed. No retries permitted until 2026-08-05 16:22:06.333570327 +0000 UTC m=+41.490416207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0dd9c938-5048-43f1-ab25-4898652787c9-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-cjmqg" (UID: "0dd9c938-5048-43f1-ab25-4898652787c9") : secret "samples-operator-tls" not found Aug 05 16:22:05.833702 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.833611 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb6wq\" (UniqueName: \"kubernetes.io/projected/791538ee-cf4d-4198-aed7-9ac30d92460e-kube-api-access-rb6wq\") pod \"service-ca-operator-784f97bcf6-stpfk\" (UID: \"791538ee-cf4d-4198-aed7-9ac30d92460e\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-stpfk" Aug 05 16:22:05.833702 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.833638 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/773c2ff3-528d-4d42-a88c-03331f44791c-metrics-certs\") pod \"router-default-cbf86bfcb-8pv7r\" (UID: \"773c2ff3-528d-4d42-a88c-03331f44791c\") " pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:05.833702 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.833666 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdnv4\" (UniqueName: \"kubernetes.io/projected/0dd9c938-5048-43f1-ab25-4898652787c9-kube-api-access-cdnv4\") pod \"cluster-samples-operator-559cbf5745-cjmqg\" (UID: \"0dd9c938-5048-43f1-ab25-4898652787c9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cjmqg" Aug 05 16:22:05.833948 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.833715 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/773c2ff3-528d-4d42-a88c-03331f44791c-service-ca-bundle\") pod \"router-default-cbf86bfcb-8pv7r\" (UID: \"773c2ff3-528d-4d42-a88c-03331f44791c\") " pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:05.834137 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.834117 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/791538ee-cf4d-4198-aed7-9ac30d92460e-config\") pod \"service-ca-operator-784f97bcf6-stpfk\" (UID: \"791538ee-cf4d-4198-aed7-9ac30d92460e\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-stpfk" Aug 05 16:22:05.835689 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.835671 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/791538ee-cf4d-4198-aed7-9ac30d92460e-serving-cert\") pod \"service-ca-operator-784f97bcf6-stpfk\" (UID: \"791538ee-cf4d-4198-aed7-9ac30d92460e\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-stpfk" Aug 05 16:22:05.850483 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.850462 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb6wq\" (UniqueName: \"kubernetes.io/projected/791538ee-cf4d-4198-aed7-9ac30d92460e-kube-api-access-rb6wq\") pod \"service-ca-operator-784f97bcf6-stpfk\" (UID: \"791538ee-cf4d-4198-aed7-9ac30d92460e\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-stpfk" Aug 05 16:22:05.851143 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.851112 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdnv4\" (UniqueName: \"kubernetes.io/projected/0dd9c938-5048-43f1-ab25-4898652787c9-kube-api-access-cdnv4\") pod \"cluster-samples-operator-559cbf5745-cjmqg\" (UID: \"0dd9c938-5048-43f1-ab25-4898652787c9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cjmqg" Aug 05 16:22:05.888081 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.888046 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:05.911701 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.911673 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-stpfk" Aug 05 16:22:05.934135 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.934102 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/773c2ff3-528d-4d42-a88c-03331f44791c-stats-auth\") pod \"router-default-cbf86bfcb-8pv7r\" (UID: \"773c2ff3-528d-4d42-a88c-03331f44791c\") " pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:05.934290 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.934193 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/773c2ff3-528d-4d42-a88c-03331f44791c-default-certificate\") pod \"router-default-cbf86bfcb-8pv7r\" (UID: \"773c2ff3-528d-4d42-a88c-03331f44791c\") " pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:05.934290 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.934222 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/773c2ff3-528d-4d42-a88c-03331f44791c-metrics-certs\") pod \"router-default-cbf86bfcb-8pv7r\" (UID: \"773c2ff3-528d-4d42-a88c-03331f44791c\") " pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:05.934290 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.934251 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/773c2ff3-528d-4d42-a88c-03331f44791c-service-ca-bundle\") pod \"router-default-cbf86bfcb-8pv7r\" (UID: \"773c2ff3-528d-4d42-a88c-03331f44791c\") " pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:05.934465 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:05.934337 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 05 16:22:05.934465 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:05.934398 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/773c2ff3-528d-4d42-a88c-03331f44791c-service-ca-bundle podName:773c2ff3-528d-4d42-a88c-03331f44791c nodeName:}" failed. No retries permitted until 2026-08-05 16:22:06.434364809 +0000 UTC m=+41.591210692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/773c2ff3-528d-4d42-a88c-03331f44791c-service-ca-bundle") pod "router-default-cbf86bfcb-8pv7r" (UID: "773c2ff3-528d-4d42-a88c-03331f44791c") : configmap references non-existent config key: service-ca.crt Aug 05 16:22:05.934465 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:05.934418 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/773c2ff3-528d-4d42-a88c-03331f44791c-metrics-certs podName:773c2ff3-528d-4d42-a88c-03331f44791c nodeName:}" failed. No retries permitted until 2026-08-05 16:22:06.43440876 +0000 UTC m=+41.591254639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/773c2ff3-528d-4d42-a88c-03331f44791c-metrics-certs") pod "router-default-cbf86bfcb-8pv7r" (UID: "773c2ff3-528d-4d42-a88c-03331f44791c") : secret "router-metrics-certs-default" not found Aug 05 16:22:05.934465 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.934434 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk72j\" (UniqueName: \"kubernetes.io/projected/773c2ff3-528d-4d42-a88c-03331f44791c-kube-api-access-zk72j\") pod \"router-default-cbf86bfcb-8pv7r\" (UID: \"773c2ff3-528d-4d42-a88c-03331f44791c\") " pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:05.936719 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.936696 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/773c2ff3-528d-4d42-a88c-03331f44791c-default-certificate\") pod \"router-default-cbf86bfcb-8pv7r\" (UID: \"773c2ff3-528d-4d42-a88c-03331f44791c\") " pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:05.948000 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.947980 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/773c2ff3-528d-4d42-a88c-03331f44791c-stats-auth\") pod \"router-default-cbf86bfcb-8pv7r\" (UID: \"773c2ff3-528d-4d42-a88c-03331f44791c\") " pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:05.958025 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.957879 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk72j\" (UniqueName: \"kubernetes.io/projected/773c2ff3-528d-4d42-a88c-03331f44791c-kube-api-access-zk72j\") pod \"router-default-cbf86bfcb-8pv7r\" (UID: \"773c2ff3-528d-4d42-a88c-03331f44791c\") " pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:05.959947 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:05.959926 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-8vzp6"] Aug 05 16:22:05.962808 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:22:05.962788 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e863683_b5a0_47e6_9d12_4a8539c943f7.slice/crio-83ee25d8817111fb869868316978c6c72e858e6dd73781363fb54afd28bfe3f6 WatchSource:0}: Error finding container 83ee25d8817111fb869868316978c6c72e858e6dd73781363fb54afd28bfe3f6: Status 404 returned error can't find the container with id 83ee25d8817111fb869868316978c6c72e858e6dd73781363fb54afd28bfe3f6 Aug 05 16:22:06.062242 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:06.062204 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:06.077057 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:06.077032 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-stpfk"] Aug 05 16:22:06.080296 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:22:06.080259 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod791538ee_cf4d_4198_aed7_9ac30d92460e.slice/crio-ae252b045df28ca7fa0edde3852f35146fbbd5b98906d3d56ef182c1105a8430 WatchSource:0}: Error finding container ae252b045df28ca7fa0edde3852f35146fbbd5b98906d3d56ef182c1105a8430: Status 404 returned error can't find the container with id ae252b045df28ca7fa0edde3852f35146fbbd5b98906d3d56ef182c1105a8430 Aug 05 16:22:06.212968 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:06.212934 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:06.336689 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:06.336652 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dd9c938-5048-43f1-ab25-4898652787c9-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-cjmqg\" (UID: \"0dd9c938-5048-43f1-ab25-4898652787c9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cjmqg" Aug 05 16:22:06.336851 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:06.336821 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 05 16:22:06.336890 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:06.336885 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dd9c938-5048-43f1-ab25-4898652787c9-samples-operator-tls podName:0dd9c938-5048-43f1-ab25-4898652787c9 nodeName:}" failed. No retries permitted until 2026-08-05 16:22:07.336869012 +0000 UTC m=+42.493714891 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0dd9c938-5048-43f1-ab25-4898652787c9-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-cjmqg" (UID: "0dd9c938-5048-43f1-ab25-4898652787c9") : secret "samples-operator-tls" not found Aug 05 16:22:06.437302 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:06.437266 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/773c2ff3-528d-4d42-a88c-03331f44791c-metrics-certs\") pod \"router-default-cbf86bfcb-8pv7r\" (UID: \"773c2ff3-528d-4d42-a88c-03331f44791c\") " pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:06.437302 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:06.437306 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/773c2ff3-528d-4d42-a88c-03331f44791c-service-ca-bundle\") pod \"router-default-cbf86bfcb-8pv7r\" (UID: \"773c2ff3-528d-4d42-a88c-03331f44791c\") " pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:06.437527 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:06.437427 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 05 16:22:06.437527 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:06.437464 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/773c2ff3-528d-4d42-a88c-03331f44791c-service-ca-bundle podName:773c2ff3-528d-4d42-a88c-03331f44791c nodeName:}" failed. No retries permitted until 2026-08-05 16:22:07.437444787 +0000 UTC m=+42.594290671 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/773c2ff3-528d-4d42-a88c-03331f44791c-service-ca-bundle") pod "router-default-cbf86bfcb-8pv7r" (UID: "773c2ff3-528d-4d42-a88c-03331f44791c") : configmap references non-existent config key: service-ca.crt Aug 05 16:22:06.437527 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:06.437485 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/773c2ff3-528d-4d42-a88c-03331f44791c-metrics-certs podName:773c2ff3-528d-4d42-a88c-03331f44791c nodeName:}" failed. No retries permitted until 2026-08-05 16:22:07.437476433 +0000 UTC m=+42.594322313 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/773c2ff3-528d-4d42-a88c-03331f44791c-metrics-certs") pod "router-default-cbf86bfcb-8pv7r" (UID: "773c2ff3-528d-4d42-a88c-03331f44791c") : secret "router-metrics-certs-default" not found Aug 05 16:22:06.478146 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:06.478105 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-stpfk" event={"ID":"791538ee-cf4d-4198-aed7-9ac30d92460e","Type":"ContainerStarted","Data":"ae252b045df28ca7fa0edde3852f35146fbbd5b98906d3d56ef182c1105a8430"} Aug 05 16:22:06.479240 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:06.479217 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-8vzp6" event={"ID":"0e863683-b5a0-47e6-9d12-4a8539c943f7","Type":"ContainerStarted","Data":"83ee25d8817111fb869868316978c6c72e858e6dd73781363fb54afd28bfe3f6"} Aug 05 16:22:07.294652 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:07.294606 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:07.343060 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:07.343034 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dd9c938-5048-43f1-ab25-4898652787c9-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-cjmqg\" (UID: \"0dd9c938-5048-43f1-ab25-4898652787c9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cjmqg" Aug 05 16:22:07.343222 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:07.343194 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 05 16:22:07.343333 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:07.343259 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dd9c938-5048-43f1-ab25-4898652787c9-samples-operator-tls podName:0dd9c938-5048-43f1-ab25-4898652787c9 nodeName:}" failed. No retries permitted until 2026-08-05 16:22:09.343242154 +0000 UTC m=+44.500088036 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0dd9c938-5048-43f1-ab25-4898652787c9-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-cjmqg" (UID: "0dd9c938-5048-43f1-ab25-4898652787c9") : secret "samples-operator-tls" not found Aug 05 16:22:07.399809 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:07.399782 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:07.444295 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:07.444268 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/773c2ff3-528d-4d42-a88c-03331f44791c-metrics-certs\") pod \"router-default-cbf86bfcb-8pv7r\" (UID: \"773c2ff3-528d-4d42-a88c-03331f44791c\") " pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:07.444482 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:07.444301 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/773c2ff3-528d-4d42-a88c-03331f44791c-service-ca-bundle\") pod \"router-default-cbf86bfcb-8pv7r\" (UID: \"773c2ff3-528d-4d42-a88c-03331f44791c\") " pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:07.444482 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:07.444439 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 05 16:22:07.444482 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:07.444441 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/773c2ff3-528d-4d42-a88c-03331f44791c-service-ca-bundle podName:773c2ff3-528d-4d42-a88c-03331f44791c nodeName:}" failed. No retries permitted until 2026-08-05 16:22:09.444424041 +0000 UTC m=+44.601269923 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/773c2ff3-528d-4d42-a88c-03331f44791c-service-ca-bundle") pod "router-default-cbf86bfcb-8pv7r" (UID: "773c2ff3-528d-4d42-a88c-03331f44791c") : configmap references non-existent config key: service-ca.crt Aug 05 16:22:07.444615 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:07.444507 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/773c2ff3-528d-4d42-a88c-03331f44791c-metrics-certs podName:773c2ff3-528d-4d42-a88c-03331f44791c nodeName:}" failed. No retries permitted until 2026-08-05 16:22:09.444489552 +0000 UTC m=+44.601335433 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/773c2ff3-528d-4d42-a88c-03331f44791c-metrics-certs") pod "router-default-cbf86bfcb-8pv7r" (UID: "773c2ff3-528d-4d42-a88c-03331f44791c") : secret "router-metrics-certs-default" not found Aug 05 16:22:07.483442 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:07.483407 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-8vzp6" event={"ID":"0e863683-b5a0-47e6-9d12-4a8539c943f7","Type":"ContainerStarted","Data":"7037853eb6cac4ace7303ca9446dec39ad839da1430d2eff2f519ad1ae3cacb6"} Aug 05 16:22:07.503016 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:07.502960 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-8vzp6" podStartSLOduration=1.147869038 podStartE2EDuration="2.502937602s" totalImagesPullingTime="1.355068564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:22:07.502253671 +0000 UTC m=+42.659099572" watchObservedRunningTime="2026-08-05 16:22:07.502937602 +0000 UTC m=+42.659783504" Aug 05 16:22:07.622603 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:07.622527 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:07.685336 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:07.685297 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:09.057543 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:09.057498 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-metrics-tls\") pod \"dns-default-w8666\" (UID: \"89e8400b-bfeb-44c4-8302-59ab47d1b4cf\") " pod="openshift-dns/dns-default-w8666" Aug 05 16:22:09.057925 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:09.057657 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 16:22:09.057925 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:09.057718 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-metrics-tls podName:89e8400b-bfeb-44c4-8302-59ab47d1b4cf nodeName:}" failed. No retries permitted until 2026-08-05 16:22:17.057702102 +0000 UTC m=+52.214547980 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-metrics-tls") pod "dns-default-w8666" (UID: "89e8400b-bfeb-44c4-8302-59ab47d1b4cf") : secret "dns-default-metrics-tls" not found Aug 05 16:22:09.359296 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:09.359219 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dd9c938-5048-43f1-ab25-4898652787c9-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-cjmqg\" (UID: \"0dd9c938-5048-43f1-ab25-4898652787c9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cjmqg" Aug 05 16:22:09.359483 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:09.359388 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 05 16:22:09.359483 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:09.359448 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dd9c938-5048-43f1-ab25-4898652787c9-samples-operator-tls podName:0dd9c938-5048-43f1-ab25-4898652787c9 nodeName:}" failed. No retries permitted until 2026-08-05 16:22:13.359433927 +0000 UTC m=+48.516279810 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0dd9c938-5048-43f1-ab25-4898652787c9-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-cjmqg" (UID: "0dd9c938-5048-43f1-ab25-4898652787c9") : secret "samples-operator-tls" not found Aug 05 16:22:09.460327 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:09.460290 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/773c2ff3-528d-4d42-a88c-03331f44791c-service-ca-bundle\") pod \"router-default-cbf86bfcb-8pv7r\" (UID: \"773c2ff3-528d-4d42-a88c-03331f44791c\") " pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:09.460531 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:09.460410 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/773c2ff3-528d-4d42-a88c-03331f44791c-metrics-certs\") pod \"router-default-cbf86bfcb-8pv7r\" (UID: \"773c2ff3-528d-4d42-a88c-03331f44791c\") " pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:09.460531 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:09.460492 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 05 16:22:09.460531 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:09.460497 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/773c2ff3-528d-4d42-a88c-03331f44791c-service-ca-bundle podName:773c2ff3-528d-4d42-a88c-03331f44791c nodeName:}" failed. No retries permitted until 2026-08-05 16:22:13.460475548 +0000 UTC m=+48.617321439 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/773c2ff3-528d-4d42-a88c-03331f44791c-service-ca-bundle") pod "router-default-cbf86bfcb-8pv7r" (UID: "773c2ff3-528d-4d42-a88c-03331f44791c") : configmap references non-existent config key: service-ca.crt Aug 05 16:22:09.460531 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:09.460532 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/773c2ff3-528d-4d42-a88c-03331f44791c-metrics-certs podName:773c2ff3-528d-4d42-a88c-03331f44791c nodeName:}" failed. No retries permitted until 2026-08-05 16:22:13.460520271 +0000 UTC m=+48.617366149 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/773c2ff3-528d-4d42-a88c-03331f44791c-metrics-certs") pod "router-default-cbf86bfcb-8pv7r" (UID: "773c2ff3-528d-4d42-a88c-03331f44791c") : secret "router-metrics-certs-default" not found Aug 05 16:22:09.489387 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:09.489354 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-stpfk" event={"ID":"791538ee-cf4d-4198-aed7-9ac30d92460e","Type":"ContainerStarted","Data":"32608dbde2c0ca3fb7f6980d9e7f4dd332401684212e83efb246be68c59ab581"} Aug 05 16:22:09.509614 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:09.509559 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-stpfk" podStartSLOduration=2.089797412 podStartE2EDuration="4.509535928s" totalImagesPullingTime="2.419738516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:22:09.509444014 +0000 UTC m=+44.666289916" watchObservedRunningTime="2026-08-05 16:22:09.509535928 +0000 UTC m=+44.666381828" Aug 05 16:22:10.015128 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:10.015094 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:10.111939 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:10.111907 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:10.260394 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:10.260353 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:10.363194 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:10.363122 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:10.720976 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:10.720909 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:10.824537 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:10.824505 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:11.033676 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:11.033643 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-srrzf"] Aug 05 16:22:11.048681 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:11.048649 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:11.056722 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:11.056702 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-srrzf" Aug 05 16:22:11.063257 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:11.063239 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 05 16:22:11.063257 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:11.063250 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 05 16:22:11.063407 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:11.063293 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 05 16:22:11.065447 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:11.065433 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 05 16:22:11.068706 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:11.068690 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rq9ls\"" Aug 05 16:22:11.070796 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:11.070779 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-srrzf"] Aug 05 16:22:11.074635 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:11.074612 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b42b84a0-641e-4650-a610-d2afa0f9dff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srrzf\" (UID: \"b42b84a0-641e-4650-a610-d2afa0f9dff1\") " pod="openshift-insights/insights-runtime-extractor-srrzf" Aug 05 16:22:11.074731 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:11.074707 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b42b84a0-641e-4650-a610-d2afa0f9dff1-data-volume\") pod \"insights-runtime-extractor-srrzf\" (UID: \"b42b84a0-641e-4650-a610-d2afa0f9dff1\") " pod="openshift-insights/insights-runtime-extractor-srrzf" Aug 05 16:22:11.074811 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:11.074797 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b42b84a0-641e-4650-a610-d2afa0f9dff1-crio-socket\") pod \"insights-runtime-extractor-srrzf\" (UID: \"b42b84a0-641e-4650-a610-d2afa0f9dff1\") " pod="openshift-insights/insights-runtime-extractor-srrzf" Aug 05 16:22:11.074873 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:11.074846 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b42b84a0-641e-4650-a610-d2afa0f9dff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srrzf\" (UID: \"b42b84a0-641e-4650-a610-d2afa0f9dff1\") " pod="openshift-insights/insights-runtime-extractor-srrzf" Aug 05 16:22:11.074922 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:11.074874 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p59jc\" (UniqueName: \"kubernetes.io/projected/b42b84a0-641e-4650-a610-d2afa0f9dff1-kube-api-access-p59jc\") pod \"insights-runtime-extractor-srrzf\" (UID: \"b42b84a0-641e-4650-a610-d2afa0f9dff1\") " pod="openshift-insights/insights-runtime-extractor-srrzf" Aug 05 16:22:11.159603 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:11.159564 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:11.175733 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:11.175704 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b42b84a0-641e-4650-a610-d2afa0f9dff1-data-volume\") pod \"insights-runtime-extractor-srrzf\" (UID: \"b42b84a0-641e-4650-a610-d2afa0f9dff1\") " pod="openshift-insights/insights-runtime-extractor-srrzf" Aug 05 16:22:11.175860 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:11.175764 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b42b84a0-641e-4650-a610-d2afa0f9dff1-crio-socket\") pod \"insights-runtime-extractor-srrzf\" (UID: \"b42b84a0-641e-4650-a610-d2afa0f9dff1\") " pod="openshift-insights/insights-runtime-extractor-srrzf" Aug 05 16:22:11.175860 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:11.175792 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b42b84a0-641e-4650-a610-d2afa0f9dff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srrzf\" (UID: \"b42b84a0-641e-4650-a610-d2afa0f9dff1\") " pod="openshift-insights/insights-runtime-extractor-srrzf" Aug 05 16:22:11.175860 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:11.175812 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p59jc\" (UniqueName: \"kubernetes.io/projected/b42b84a0-641e-4650-a610-d2afa0f9dff1-kube-api-access-p59jc\") pod \"insights-runtime-extractor-srrzf\" (UID: \"b42b84a0-641e-4650-a610-d2afa0f9dff1\") " pod="openshift-insights/insights-runtime-extractor-srrzf" Aug 05 16:22:11.175860 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:11.175835 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b42b84a0-641e-4650-a610-d2afa0f9dff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srrzf\" (UID: \"b42b84a0-641e-4650-a610-d2afa0f9dff1\") " pod="openshift-insights/insights-runtime-extractor-srrzf" Aug 05 16:22:11.176028 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:11.175898 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 16:22:11.176028 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:11.175929 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 16:22:11.176028 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:11.175970 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b42b84a0-641e-4650-a610-d2afa0f9dff1-kube-rbac-proxy-cm podName:b42b84a0-641e-4650-a610-d2afa0f9dff1 nodeName:}" failed. No retries permitted until 2026-08-05 16:22:11.675950763 +0000 UTC m=+46.832796726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b42b84a0-641e-4650-a610-d2afa0f9dff1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-srrzf" (UID: "b42b84a0-641e-4650-a610-d2afa0f9dff1") : configmap "kube-rbac-proxy" not found Aug 05 16:22:11.176028 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:11.175998 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b42b84a0-641e-4650-a610-d2afa0f9dff1-insights-runtime-extractor-tls podName:b42b84a0-641e-4650-a610-d2afa0f9dff1 nodeName:}" failed. No retries permitted until 2026-08-05 16:22:11.675980151 +0000 UTC m=+46.832826048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b42b84a0-641e-4650-a610-d2afa0f9dff1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-srrzf" (UID: "b42b84a0-641e-4650-a610-d2afa0f9dff1") : secret "insights-runtime-extractor-tls" not found Aug 05 16:22:11.176190 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:11.176054 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b42b84a0-641e-4650-a610-d2afa0f9dff1-crio-socket\") pod \"insights-runtime-extractor-srrzf\" (UID: \"b42b84a0-641e-4650-a610-d2afa0f9dff1\") " pod="openshift-insights/insights-runtime-extractor-srrzf" Aug 05 16:22:11.176190 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:11.176060 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b42b84a0-641e-4650-a610-d2afa0f9dff1-data-volume\") pod \"insights-runtime-extractor-srrzf\" (UID: \"b42b84a0-641e-4650-a610-d2afa0f9dff1\") " pod="openshift-insights/insights-runtime-extractor-srrzf" Aug 05 16:22:11.185588 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:11.185562 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p59jc\" (UniqueName: \"kubernetes.io/projected/b42b84a0-641e-4650-a610-d2afa0f9dff1-kube-api-access-p59jc\") pod \"insights-runtime-extractor-srrzf\" (UID: \"b42b84a0-641e-4650-a610-d2afa0f9dff1\") " pod="openshift-insights/insights-runtime-extractor-srrzf" Aug 05 16:22:11.680663 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:11.680629 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b42b84a0-641e-4650-a610-d2afa0f9dff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srrzf\" (UID: \"b42b84a0-641e-4650-a610-d2afa0f9dff1\") " pod="openshift-insights/insights-runtime-extractor-srrzf" Aug 05 16:22:11.680663 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:11.680670 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b42b84a0-641e-4650-a610-d2afa0f9dff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srrzf\" (UID: \"b42b84a0-641e-4650-a610-d2afa0f9dff1\") " pod="openshift-insights/insights-runtime-extractor-srrzf" Aug 05 16:22:11.680853 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:11.680789 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 16:22:11.680853 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:11.680840 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b42b84a0-641e-4650-a610-d2afa0f9dff1-kube-rbac-proxy-cm podName:b42b84a0-641e-4650-a610-d2afa0f9dff1 nodeName:}" failed. No retries permitted until 2026-08-05 16:22:12.680827199 +0000 UTC m=+47.837673077 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b42b84a0-641e-4650-a610-d2afa0f9dff1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-srrzf" (UID: "b42b84a0-641e-4650-a610-d2afa0f9dff1") : configmap "kube-rbac-proxy" not found Aug 05 16:22:11.680917 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:11.680787 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 16:22:11.680954 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:11.680930 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b42b84a0-641e-4650-a610-d2afa0f9dff1-insights-runtime-extractor-tls podName:b42b84a0-641e-4650-a610-d2afa0f9dff1 nodeName:}" failed. No retries permitted until 2026-08-05 16:22:12.680914601 +0000 UTC m=+47.837760482 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b42b84a0-641e-4650-a610-d2afa0f9dff1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-srrzf" (UID: "b42b84a0-641e-4650-a610-d2afa0f9dff1") : secret "insights-runtime-extractor-tls" not found Aug 05 16:22:12.131774 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:12.131741 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:12.248012 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:12.247976 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:12.341935 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:12.341900 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-fvwpw"] Aug 05 16:22:12.343701 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:12.343679 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:12.360638 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:12.360609 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-fvwpw"] Aug 05 16:22:12.360741 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:12.360719 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-fvwpw" Aug 05 16:22:12.364103 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:12.364084 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Aug 05 16:22:12.364205 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:12.364118 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Aug 05 16:22:12.365452 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:12.365438 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-khct5\"" Aug 05 16:22:12.365601 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:12.365482 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Aug 05 16:22:12.365601 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:12.365482 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Aug 05 16:22:12.387321 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:12.387261 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58jlc\" (UniqueName: \"kubernetes.io/projected/11049624-886c-4eb8-a200-54fa7593e6c1-kube-api-access-58jlc\") pod \"service-ca-98c88fbb5-fvwpw\" (UID: \"11049624-886c-4eb8-a200-54fa7593e6c1\") " pod="openshift-service-ca/service-ca-98c88fbb5-fvwpw" Aug 05 16:22:12.387321 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:12.387306 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/11049624-886c-4eb8-a200-54fa7593e6c1-signing-cabundle\") pod \"service-ca-98c88fbb5-fvwpw\" (UID: \"11049624-886c-4eb8-a200-54fa7593e6c1\") " pod="openshift-service-ca/service-ca-98c88fbb5-fvwpw" Aug 05 16:22:12.387491 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:12.387329 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/11049624-886c-4eb8-a200-54fa7593e6c1-signing-key\") pod \"service-ca-98c88fbb5-fvwpw\" (UID: \"11049624-886c-4eb8-a200-54fa7593e6c1\") " pod="openshift-service-ca/service-ca-98c88fbb5-fvwpw" Aug 05 16:22:12.406857 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:12.406837 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:12.488657 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:12.488623 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58jlc\" (UniqueName: \"kubernetes.io/projected/11049624-886c-4eb8-a200-54fa7593e6c1-kube-api-access-58jlc\") pod \"service-ca-98c88fbb5-fvwpw\" (UID: \"11049624-886c-4eb8-a200-54fa7593e6c1\") " pod="openshift-service-ca/service-ca-98c88fbb5-fvwpw" Aug 05 16:22:12.488812 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:12.488671 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/11049624-886c-4eb8-a200-54fa7593e6c1-signing-cabundle\") pod \"service-ca-98c88fbb5-fvwpw\" (UID: \"11049624-886c-4eb8-a200-54fa7593e6c1\") " pod="openshift-service-ca/service-ca-98c88fbb5-fvwpw" Aug 05 16:22:12.488812 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:12.488695 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/11049624-886c-4eb8-a200-54fa7593e6c1-signing-key\") pod \"service-ca-98c88fbb5-fvwpw\" (UID: \"11049624-886c-4eb8-a200-54fa7593e6c1\") " pod="openshift-service-ca/service-ca-98c88fbb5-fvwpw" Aug 05 16:22:12.489318 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:12.489296 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/11049624-886c-4eb8-a200-54fa7593e6c1-signing-cabundle\") pod \"service-ca-98c88fbb5-fvwpw\" (UID: \"11049624-886c-4eb8-a200-54fa7593e6c1\") " pod="openshift-service-ca/service-ca-98c88fbb5-fvwpw" Aug 05 16:22:12.490961 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:12.490935 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/11049624-886c-4eb8-a200-54fa7593e6c1-signing-key\") pod \"service-ca-98c88fbb5-fvwpw\" (UID: \"11049624-886c-4eb8-a200-54fa7593e6c1\") " pod="openshift-service-ca/service-ca-98c88fbb5-fvwpw" Aug 05 16:22:12.497525 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:12.497497 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58jlc\" (UniqueName: \"kubernetes.io/projected/11049624-886c-4eb8-a200-54fa7593e6c1-kube-api-access-58jlc\") pod \"service-ca-98c88fbb5-fvwpw\" (UID: \"11049624-886c-4eb8-a200-54fa7593e6c1\") " pod="openshift-service-ca/service-ca-98c88fbb5-fvwpw" Aug 05 16:22:12.669994 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:12.669914 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-fvwpw" Aug 05 16:22:12.690581 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:12.690555 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b42b84a0-641e-4650-a610-d2afa0f9dff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srrzf\" (UID: \"b42b84a0-641e-4650-a610-d2afa0f9dff1\") " pod="openshift-insights/insights-runtime-extractor-srrzf" Aug 05 16:22:12.690715 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:12.690679 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 16:22:12.690774 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:12.690763 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b42b84a0-641e-4650-a610-d2afa0f9dff1-kube-rbac-proxy-cm podName:b42b84a0-641e-4650-a610-d2afa0f9dff1 nodeName:}" failed. No retries permitted until 2026-08-05 16:22:14.690744167 +0000 UTC m=+49.847590071 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b42b84a0-641e-4650-a610-d2afa0f9dff1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-srrzf" (UID: "b42b84a0-641e-4650-a610-d2afa0f9dff1") : configmap "kube-rbac-proxy" not found Aug 05 16:22:12.690817 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:12.690792 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b42b84a0-641e-4650-a610-d2afa0f9dff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srrzf\" (UID: \"b42b84a0-641e-4650-a610-d2afa0f9dff1\") " pod="openshift-insights/insights-runtime-extractor-srrzf" Aug 05 16:22:12.690900 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:12.690881 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 16:22:12.691032 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:12.690922 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b42b84a0-641e-4650-a610-d2afa0f9dff1-insights-runtime-extractor-tls podName:b42b84a0-641e-4650-a610-d2afa0f9dff1 nodeName:}" failed. No retries permitted until 2026-08-05 16:22:14.69091365 +0000 UTC m=+49.847759529 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b42b84a0-641e-4650-a610-d2afa0f9dff1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-srrzf" (UID: "b42b84a0-641e-4650-a610-d2afa0f9dff1") : secret "insights-runtime-extractor-tls" not found Aug 05 16:22:12.817973 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:12.817942 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-fvwpw"] Aug 05 16:22:12.820990 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:22:12.820957 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11049624_886c_4eb8_a200_54fa7593e6c1.slice/crio-6aad7d43a49d58a984b59dff921747428213d3d04248448ec6d7375c27915800 WatchSource:0}: Error finding container 6aad7d43a49d58a984b59dff921747428213d3d04248448ec6d7375c27915800: Status 404 returned error can't find the container with id 6aad7d43a49d58a984b59dff921747428213d3d04248448ec6d7375c27915800 Aug 05 16:22:13.177560 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:13.177481 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vbh4x_fe314431-1429-4319-b3a0-0c7287575cef/dns-node-resolver/0.log" Aug 05 16:22:13.346289 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:13.346256 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:13.396837 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:13.396807 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dd9c938-5048-43f1-ab25-4898652787c9-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-cjmqg\" (UID: \"0dd9c938-5048-43f1-ab25-4898652787c9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cjmqg" Aug 05 16:22:13.396973 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:13.396962 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 05 16:22:13.397043 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:13.397032 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dd9c938-5048-43f1-ab25-4898652787c9-samples-operator-tls podName:0dd9c938-5048-43f1-ab25-4898652787c9 nodeName:}" failed. No retries permitted until 2026-08-05 16:22:21.397015143 +0000 UTC m=+56.553861025 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0dd9c938-5048-43f1-ab25-4898652787c9-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-cjmqg" (UID: "0dd9c938-5048-43f1-ab25-4898652787c9") : secret "samples-operator-tls" not found Aug 05 16:22:13.497911 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:13.497829 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/773c2ff3-528d-4d42-a88c-03331f44791c-metrics-certs\") pod \"router-default-cbf86bfcb-8pv7r\" (UID: \"773c2ff3-528d-4d42-a88c-03331f44791c\") " pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:13.497911 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:13.497873 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/773c2ff3-528d-4d42-a88c-03331f44791c-service-ca-bundle\") pod \"router-default-cbf86bfcb-8pv7r\" (UID: \"773c2ff3-528d-4d42-a88c-03331f44791c\") " pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:13.498100 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:13.497996 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 05 16:22:13.498100 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:13.498058 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/773c2ff3-528d-4d42-a88c-03331f44791c-service-ca-bundle podName:773c2ff3-528d-4d42-a88c-03331f44791c nodeName:}" failed. No retries permitted until 2026-08-05 16:22:21.498034321 +0000 UTC m=+56.654880223 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/773c2ff3-528d-4d42-a88c-03331f44791c-service-ca-bundle") pod "router-default-cbf86bfcb-8pv7r" (UID: "773c2ff3-528d-4d42-a88c-03331f44791c") : configmap references non-existent config key: service-ca.crt Aug 05 16:22:13.498100 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:13.498080 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/773c2ff3-528d-4d42-a88c-03331f44791c-metrics-certs podName:773c2ff3-528d-4d42-a88c-03331f44791c nodeName:}" failed. No retries permitted until 2026-08-05 16:22:21.498070392 +0000 UTC m=+56.654916277 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/773c2ff3-528d-4d42-a88c-03331f44791c-metrics-certs") pod "router-default-cbf86bfcb-8pv7r" (UID: "773c2ff3-528d-4d42-a88c-03331f44791c") : secret "router-metrics-certs-default" not found Aug 05 16:22:13.501333 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:13.501302 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-fvwpw" event={"ID":"11049624-886c-4eb8-a200-54fa7593e6c1","Type":"ContainerStarted","Data":"3cbf849640501aef904188a4ddd9ad0e34c4e8a3992b0c3a2ab4122870b0ea0d"} Aug 05 16:22:13.501458 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:13.501352 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-fvwpw" event={"ID":"11049624-886c-4eb8-a200-54fa7593e6c1","Type":"ContainerStarted","Data":"6aad7d43a49d58a984b59dff921747428213d3d04248448ec6d7375c27915800"} Aug 05 16:22:13.532199 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:13.532154 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-98c88fbb5-fvwpw" podStartSLOduration=1.532138229 podStartE2EDuration="1.532138229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:22:13.531409694 +0000 UTC m=+48.688255596" watchObservedRunningTime="2026-08-05 16:22:13.532138229 +0000 UTC m=+48.688984208" Aug 05 16:22:13.778623 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:13.778597 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bmvw9_f2a0163b-3977-4a52-8c23-6139f01a9b2d/node-ca/0.log" Aug 05 16:22:14.284748 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:14.284714 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:14.390860 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:14.384948 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:14.537844 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:14.537754 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:14.664260 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:14.664226 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:14.706896 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:14.706847 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b42b84a0-641e-4650-a610-d2afa0f9dff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srrzf\" (UID: \"b42b84a0-641e-4650-a610-d2afa0f9dff1\") " pod="openshift-insights/insights-runtime-extractor-srrzf" Aug 05 16:22:14.707084 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:14.706907 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b42b84a0-641e-4650-a610-d2afa0f9dff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srrzf\" (UID: \"b42b84a0-641e-4650-a610-d2afa0f9dff1\") " pod="openshift-insights/insights-runtime-extractor-srrzf" Aug 05 16:22:14.707084 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:14.706998 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 16:22:14.707084 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:14.707011 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 16:22:14.707084 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:14.707071 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b42b84a0-641e-4650-a610-d2afa0f9dff1-insights-runtime-extractor-tls podName:b42b84a0-641e-4650-a610-d2afa0f9dff1 nodeName:}" failed. No retries permitted until 2026-08-05 16:22:18.707049929 +0000 UTC m=+53.863895810 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b42b84a0-641e-4650-a610-d2afa0f9dff1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-srrzf" (UID: "b42b84a0-641e-4650-a610-d2afa0f9dff1") : secret "insights-runtime-extractor-tls" not found Aug 05 16:22:14.707297 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:14.707092 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b42b84a0-641e-4650-a610-d2afa0f9dff1-kube-rbac-proxy-cm podName:b42b84a0-641e-4650-a610-d2afa0f9dff1 nodeName:}" failed. No retries permitted until 2026-08-05 16:22:18.707082629 +0000 UTC m=+53.863928514 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b42b84a0-641e-4650-a610-d2afa0f9dff1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-srrzf" (UID: "b42b84a0-641e-4650-a610-d2afa0f9dff1") : configmap "kube-rbac-proxy" not found Aug 05 16:22:14.768027 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:14.767970 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:14.776556 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:14.776534 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-477z7_76352a33-7d75-494f-9741-931a2cc97c40/serve-healthcheck-canary/0.log" Aug 05 16:22:15.171473 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:15.171437 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:15.574631 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:15.574593 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:16.166708 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:16.166662 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:16.566722 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:16.566688 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:16.640940 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:16.640912 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:16.998427 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:16.998318 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:17.127286 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:17.127252 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-metrics-tls\") pod \"dns-default-w8666\" (UID: \"89e8400b-bfeb-44c4-8302-59ab47d1b4cf\") " pod="openshift-dns/dns-default-w8666" Aug 05 16:22:17.127488 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:17.127428 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 16:22:17.127488 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:17.127486 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-metrics-tls podName:89e8400b-bfeb-44c4-8302-59ab47d1b4cf nodeName:}" failed. No retries permitted until 2026-08-05 16:22:33.127471085 +0000 UTC m=+68.284316964 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-metrics-tls") pod "dns-default-w8666" (UID: "89e8400b-bfeb-44c4-8302-59ab47d1b4cf") : secret "dns-default-metrics-tls" not found Aug 05 16:22:17.376063 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:17.376020 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:17.573119 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:17.573089 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:17.966233 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:17.966196 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:18.067355 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:18.067323 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:18.455823 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:18.455793 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:18.604744 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:18.604708 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:18.743155 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:18.743082 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b42b84a0-641e-4650-a610-d2afa0f9dff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srrzf\" (UID: \"b42b84a0-641e-4650-a610-d2afa0f9dff1\") " pod="openshift-insights/insights-runtime-extractor-srrzf" Aug 05 16:22:18.743155 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:18.743121 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b42b84a0-641e-4650-a610-d2afa0f9dff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srrzf\" (UID: \"b42b84a0-641e-4650-a610-d2afa0f9dff1\") " pod="openshift-insights/insights-runtime-extractor-srrzf" Aug 05 16:22:18.743345 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:18.743194 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 16:22:18.743345 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:18.743224 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 16:22:18.743345 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:18.743245 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b42b84a0-641e-4650-a610-d2afa0f9dff1-kube-rbac-proxy-cm podName:b42b84a0-641e-4650-a610-d2afa0f9dff1 nodeName:}" failed. No retries permitted until 2026-08-05 16:22:26.743231937 +0000 UTC m=+61.900077817 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b42b84a0-641e-4650-a610-d2afa0f9dff1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-srrzf" (UID: "b42b84a0-641e-4650-a610-d2afa0f9dff1") : configmap "kube-rbac-proxy" not found Aug 05 16:22:18.743345 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:22:18.743276 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b42b84a0-641e-4650-a610-d2afa0f9dff1-insights-runtime-extractor-tls podName:b42b84a0-641e-4650-a610-d2afa0f9dff1 nodeName:}" failed. No retries permitted until 2026-08-05 16:22:26.743261917 +0000 UTC m=+61.900107797 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b42b84a0-641e-4650-a610-d2afa0f9dff1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-srrzf" (UID: "b42b84a0-641e-4650-a610-d2afa0f9dff1") : secret "insights-runtime-extractor-tls" not found Aug 05 16:22:19.164752 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:19.164711 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:19.265881 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:19.265843 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:19.364710 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:19.364674 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:19.776049 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:19.776011 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:19.963989 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:19.963939 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:20.564343 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:20.564311 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:20.665348 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:20.665307 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:21.006006 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:21.005922 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:21.205269 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:21.205238 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:21.463909 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:21.463867 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dd9c938-5048-43f1-ab25-4898652787c9-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-cjmqg\" (UID: \"0dd9c938-5048-43f1-ab25-4898652787c9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cjmqg" Aug 05 16:22:21.466185 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:21.466155 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dd9c938-5048-43f1-ab25-4898652787c9-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-cjmqg\" (UID: \"0dd9c938-5048-43f1-ab25-4898652787c9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cjmqg" Aug 05 16:22:21.534756 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:21.534724 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cjmqg" Aug 05 16:22:21.564781 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:21.564745 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/773c2ff3-528d-4d42-a88c-03331f44791c-metrics-certs\") pod \"router-default-cbf86bfcb-8pv7r\" (UID: \"773c2ff3-528d-4d42-a88c-03331f44791c\") " pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:21.564781 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:21.564783 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/773c2ff3-528d-4d42-a88c-03331f44791c-service-ca-bundle\") pod \"router-default-cbf86bfcb-8pv7r\" (UID: \"773c2ff3-528d-4d42-a88c-03331f44791c\") " pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:21.565337 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:21.565315 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/773c2ff3-528d-4d42-a88c-03331f44791c-service-ca-bundle\") pod \"router-default-cbf86bfcb-8pv7r\" (UID: \"773c2ff3-528d-4d42-a88c-03331f44791c\") " pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:21.567264 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:21.567239 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/773c2ff3-528d-4d42-a88c-03331f44791c-metrics-certs\") pod \"router-default-cbf86bfcb-8pv7r\" (UID: \"773c2ff3-528d-4d42-a88c-03331f44791c\") " pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:21.656920 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:21.656891 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cjmqg"] Aug 05 16:22:21.703799 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:21.703776 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:21.776496 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:21.776469 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:21.834423 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:21.834300 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-cbf86bfcb-8pv7r"] Aug 05 16:22:21.837345 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:22:21.837319 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod773c2ff3_528d_4d42_a88c_03331f44791c.slice/crio-c450393112bd02ee7c450aadf9b82f3566b29059d0f353c2a17e2e637ebeed30 WatchSource:0}: Error finding container c450393112bd02ee7c450aadf9b82f3566b29059d0f353c2a17e2e637ebeed30: Status 404 returned error can't find the container with id c450393112bd02ee7c450aadf9b82f3566b29059d0f353c2a17e2e637ebeed30 Aug 05 16:22:22.171540 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:22.171492 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:22.527895 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:22.527856 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" event={"ID":"773c2ff3-528d-4d42-a88c-03331f44791c","Type":"ContainerStarted","Data":"3d4d09c8a352e29eaa947a2ff61daf3fd057dff5c004bd0bb872885adf2cda80"} Aug 05 16:22:22.528109 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:22.527914 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" event={"ID":"773c2ff3-528d-4d42-a88c-03331f44791c","Type":"ContainerStarted","Data":"c450393112bd02ee7c450aadf9b82f3566b29059d0f353c2a17e2e637ebeed30"} Aug 05 16:22:22.529249 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:22.529225 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cjmqg" event={"ID":"0dd9c938-5048-43f1-ab25-4898652787c9","Type":"ContainerStarted","Data":"b667de71d4eebf9b5536156054d0edb8100b392a2fa3c640bcde3c9667e1844a"} Aug 05 16:22:22.551369 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:22.551311 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" podStartSLOduration=17.551293518 podStartE2EDuration="17.551293518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:22:22.549849207 +0000 UTC m=+57.706695117" watchObservedRunningTime="2026-08-05 16:22:22.551293518 +0000 UTC m=+57.708139423" Aug 05 16:22:22.573053 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:22.573018 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:22.704566 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:22.704508 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:22.707495 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:22.707471 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:22.769318 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:22.769286 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:23.173469 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:23.173421 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:23.532309 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:23.532275 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:23.533482 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:23.533461 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-cbf86bfcb-8pv7r" Aug 05 16:22:23.614941 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:23.614893 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:24.182135 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:24.182026 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:24.284399 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:24.284343 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:24.383964 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:24.383929 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:24.483343 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:24.483240 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:24.537166 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:24.537130 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cjmqg" event={"ID":"0dd9c938-5048-43f1-ab25-4898652787c9","Type":"ContainerStarted","Data":"26706278e05e4c5afdad8b44f866e62c6e18528b90cf61db3e1a3a9aaf7cc7a6"} Aug 05 16:22:24.537323 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:24.537192 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cjmqg" event={"ID":"0dd9c938-5048-43f1-ab25-4898652787c9","Type":"ContainerStarted","Data":"864b68dc09e0c6f0f9e764eb4e96b752dcad76e06686087f887452c766729391"} Aug 05 16:22:24.560138 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:24.560093 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-cjmqg" podStartSLOduration=17.45723046 podStartE2EDuration="19.560076302s" totalImagesPullingTime="2.102845842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:22:24.559900231 +0000 UTC m=+59.716746158" watchObservedRunningTime="2026-08-05 16:22:24.560076302 +0000 UTC m=+59.716922203" Aug 05 16:22:24.568020 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:24.567992 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:24.758077 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:24.758048 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:25.161888 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:25.161797 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:25.361406 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:25.361354 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:26.808433 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:26.808389 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b42b84a0-641e-4650-a610-d2afa0f9dff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srrzf\" (UID: \"b42b84a0-641e-4650-a610-d2afa0f9dff1\") " pod="openshift-insights/insights-runtime-extractor-srrzf" Aug 05 16:22:26.808832 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:26.808449 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b42b84a0-641e-4650-a610-d2afa0f9dff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srrzf\" (UID: \"b42b84a0-641e-4650-a610-d2afa0f9dff1\") " pod="openshift-insights/insights-runtime-extractor-srrzf" Aug 05 16:22:26.809040 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:26.809018 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b42b84a0-641e-4650-a610-d2afa0f9dff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srrzf\" (UID: \"b42b84a0-641e-4650-a610-d2afa0f9dff1\") " pod="openshift-insights/insights-runtime-extractor-srrzf" Aug 05 16:22:26.810728 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:26.810709 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b42b84a0-641e-4650-a610-d2afa0f9dff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srrzf\" (UID: \"b42b84a0-641e-4650-a610-d2afa0f9dff1\") " pod="openshift-insights/insights-runtime-extractor-srrzf" Aug 05 16:22:26.968095 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:26.968063 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rq9ls\"" Aug 05 16:22:26.976083 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:26.976060 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-srrzf" Aug 05 16:22:27.099618 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:27.099549 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-srrzf"] Aug 05 16:22:27.103009 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:22:27.102981 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42b84a0_641e_4650_a610_d2afa0f9dff1.slice/crio-15c8440b9649640dd6cf7b58cb83b510c5291a9ea7eeae1330855b1a395023fd WatchSource:0}: Error finding container 15c8440b9649640dd6cf7b58cb83b510c5291a9ea7eeae1330855b1a395023fd: Status 404 returned error can't find the container with id 15c8440b9649640dd6cf7b58cb83b510c5291a9ea7eeae1330855b1a395023fd Aug 05 16:22:27.304509 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:27.304479 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:27.369975 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:27.369898 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:27.547326 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:27.547297 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-srrzf" event={"ID":"b42b84a0-641e-4650-a610-d2afa0f9dff1","Type":"ContainerStarted","Data":"8b4c43c1aed96f33cb430aedd19324180c9d0ed140ce129bef12c7f57e06848f"} Aug 05 16:22:27.547500 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:27.547350 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-srrzf" event={"ID":"b42b84a0-641e-4650-a610-d2afa0f9dff1","Type":"ContainerStarted","Data":"15c8440b9649640dd6cf7b58cb83b510c5291a9ea7eeae1330855b1a395023fd"} Aug 05 16:22:28.460875 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:28.460797 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjr22" Aug 05 16:22:28.551651 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:28.551616 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-srrzf" event={"ID":"b42b84a0-641e-4650-a610-d2afa0f9dff1","Type":"ContainerStarted","Data":"ca84c3eccfb8b09e0b9fd810a131f2a2aa5093eb52e0d5902fb0b26d241c8c37"} Aug 05 16:22:29.048583 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:29.048520 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:29.149855 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:29.149815 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:29.348554 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:29.348469 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:29.729067 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:29.728986 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:30.163310 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:30.163283 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:30.264081 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:30.264042 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:30.545598 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:30.545556 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:30.559563 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:30.559534 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-srrzf" event={"ID":"b42b84a0-641e-4650-a610-d2afa0f9dff1","Type":"ContainerStarted","Data":"4f8c9ca1628f40463322b745fae94ba228abc2733088847aa7f667f23887d049"} Aug 05 16:22:30.581781 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:30.581736 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-srrzf" podStartSLOduration=17.630557027000002 podStartE2EDuration="20.581721212s" totalImagesPullingTime="2.951164185s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:22:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:22:30.579953051 +0000 UTC m=+65.736798950" watchObservedRunningTime="2026-08-05 16:22:30.581721212 +0000 UTC m=+65.738567113" Aug 05 16:22:30.602898 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:30.602875 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:31.142927 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:31.142890 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d233df-9262-45b2-8065-0d44a7ee9e12-original-pull-secret\") pod \"global-pull-secret-syncer-qjdbg\" (UID: \"39d233df-9262-45b2-8065-0d44a7ee9e12\") " pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:22:31.142927 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:31.142932 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d-metrics-certs\") pod \"network-metrics-daemon-88qn5\" (UID: \"aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d\") " pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:22:31.146034 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:31.146017 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 05 16:22:31.146142 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:31.146017 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 05 16:22:31.155803 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:31.155779 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d233df-9262-45b2-8065-0d44a7ee9e12-original-pull-secret\") pod \"global-pull-secret-syncer-qjdbg\" (UID: \"39d233df-9262-45b2-8065-0d44a7ee9e12\") " pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:22:31.155916 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:31.155843 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d-metrics-certs\") pod \"network-metrics-daemon-88qn5\" (UID: \"aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d\") " pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:22:31.208669 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:31.208637 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:31.243750 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:31.243718 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x5w8\" (UniqueName: \"kubernetes.io/projected/e303e40d-9669-4230-a1e1-c75ea3a2c4df-kube-api-access-8x5w8\") pod \"network-check-target-pwx27\" (UID: \"e303e40d-9669-4230-a1e1-c75ea3a2c4df\") " pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:22:31.246797 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:31.246779 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 05 16:22:31.257320 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:31.257303 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 05 16:22:31.267197 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:31.267175 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x5w8\" (UniqueName: \"kubernetes.io/projected/e303e40d-9669-4230-a1e1-c75ea3a2c4df-kube-api-access-8x5w8\") pod \"network-check-target-pwx27\" (UID: \"e303e40d-9669-4230-a1e1-c75ea3a2c4df\") " pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:22:31.306620 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:31.306591 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:31.309820 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:31.309805 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjdbg" Aug 05 16:22:31.318331 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:31.318313 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w287q\"" Aug 05 16:22:31.325214 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:31.325194 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88qn5" Aug 05 16:22:31.325467 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:31.325450 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-64w4b\"" Aug 05 16:22:31.333108 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:31.333081 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:22:31.457687 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:31.457659 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qjdbg"] Aug 05 16:22:31.461850 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:22:31.461821 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d233df_9262_45b2_8065_0d44a7ee9e12.slice/crio-477cac40feff4b8943c8815f758acc86f4519e04e0f321a864d9a36c4f9157a6 WatchSource:0}: Error finding container 477cac40feff4b8943c8815f758acc86f4519e04e0f321a864d9a36c4f9157a6: Status 404 returned error can't find the container with id 477cac40feff4b8943c8815f758acc86f4519e04e0f321a864d9a36c4f9157a6 Aug 05 16:22:31.480402 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:31.480363 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-88qn5"] Aug 05 16:22:31.483836 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:22:31.483811 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad2f6f3_7db8_4b0f_add0_d6cec8cd2c0d.slice/crio-8e7b2248c92d1b6fd897fb71629d485cc80cfeebb89ea6b533aa23194d27b6a9 WatchSource:0}: Error finding container 8e7b2248c92d1b6fd897fb71629d485cc80cfeebb89ea6b533aa23194d27b6a9: Status 404 returned error can't find the container with id 8e7b2248c92d1b6fd897fb71629d485cc80cfeebb89ea6b533aa23194d27b6a9 Aug 05 16:22:31.499007 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:31.498984 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pwx27"] Aug 05 16:22:31.502108 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:22:31.502084 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode303e40d_9669_4230_a1e1_c75ea3a2c4df.slice/crio-ebb7d02d569f4f7408d4139dd0f73599c37bd3ffeefba70578cc545052d07574 WatchSource:0}: Error finding container ebb7d02d569f4f7408d4139dd0f73599c37bd3ffeefba70578cc545052d07574: Status 404 returned error can't find the container with id ebb7d02d569f4f7408d4139dd0f73599c37bd3ffeefba70578cc545052d07574 Aug 05 16:22:31.563266 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:31.563232 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pwx27" event={"ID":"e303e40d-9669-4230-a1e1-c75ea3a2c4df","Type":"ContainerStarted","Data":"ebb7d02d569f4f7408d4139dd0f73599c37bd3ffeefba70578cc545052d07574"} Aug 05 16:22:31.564233 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:31.564204 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qjdbg" event={"ID":"39d233df-9262-45b2-8065-0d44a7ee9e12","Type":"ContainerStarted","Data":"477cac40feff4b8943c8815f758acc86f4519e04e0f321a864d9a36c4f9157a6"} Aug 05 16:22:31.565108 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:31.565081 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-88qn5" event={"ID":"aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d","Type":"ContainerStarted","Data":"8e7b2248c92d1b6fd897fb71629d485cc80cfeebb89ea6b533aa23194d27b6a9"} Aug 05 16:22:32.573233 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:32.573197 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-88qn5" event={"ID":"aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d","Type":"ContainerStarted","Data":"e6d9a158ff2ac45ef497ccf3a20057b240f82da38a59baaf9baed019cdd44cef"} Aug 05 16:22:32.975716 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:32.975672 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:33.051209 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:33.051138 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:33.161058 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:33.161017 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-metrics-tls\") pod \"dns-default-w8666\" (UID: \"89e8400b-bfeb-44c4-8302-59ab47d1b4cf\") " pod="openshift-dns/dns-default-w8666" Aug 05 16:22:33.163868 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:33.163807 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89e8400b-bfeb-44c4-8302-59ab47d1b4cf-metrics-tls\") pod \"dns-default-w8666\" (UID: \"89e8400b-bfeb-44c4-8302-59ab47d1b4cf\") " pod="openshift-dns/dns-default-w8666" Aug 05 16:22:33.272593 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:33.272557 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:33.344940 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:33.344900 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:33.393338 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:33.393309 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cnw72\"" Aug 05 16:22:33.400186 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:33.400159 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w8666" Aug 05 16:22:33.563031 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:33.562655 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w8666"] Aug 05 16:22:33.565823 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:22:33.565783 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e8400b_bfeb_44c4_8302_59ab47d1b4cf.slice/crio-97b2c11cd33db1833a73d51db8747548a13d0920396e8350f3ebc228de8d3977 WatchSource:0}: Error finding container 97b2c11cd33db1833a73d51db8747548a13d0920396e8350f3ebc228de8d3977: Status 404 returned error can't find the container with id 97b2c11cd33db1833a73d51db8747548a13d0920396e8350f3ebc228de8d3977 Aug 05 16:22:33.580569 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:33.580470 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-88qn5" event={"ID":"aad2f6f3-7db8-4b0f-add0-d6cec8cd2c0d","Type":"ContainerStarted","Data":"aed9e120e3cd2324dbca8a57557e274c680180ca5419cee7ca046a4fa073dd5e"} Aug 05 16:22:33.582255 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:33.582197 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w8666" event={"ID":"89e8400b-bfeb-44c4-8302-59ab47d1b4cf","Type":"ContainerStarted","Data":"97b2c11cd33db1833a73d51db8747548a13d0920396e8350f3ebc228de8d3977"} Aug 05 16:22:33.601962 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:33.601910 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-88qn5" podStartSLOduration=67.645063931 podStartE2EDuration="1m8.601890695s" totalImagesPullingTime="956.826764ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:22:33.600617732 +0000 UTC m=+68.757463635" watchObservedRunningTime="2026-08-05 16:22:33.601890695 +0000 UTC m=+68.758736600" Aug 05 16:22:34.957507 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:34.957474 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d887b5fb-ggh9f"] Aug 05 16:22:34.960851 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:34.960826 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d887b5fb-ggh9f" Aug 05 16:22:34.965360 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:34.965341 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Aug 05 16:22:34.965645 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:34.965623 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Aug 05 16:22:34.965741 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:34.965704 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Aug 05 16:22:34.965876 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:34.965857 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Aug 05 16:22:34.967017 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:34.966829 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Aug 05 16:22:34.967017 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:34.966937 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Aug 05 16:22:34.967402 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:34.967363 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Aug 05 16:22:34.979398 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:34.979359 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d887b5fb-ggh9f"] Aug 05 16:22:35.074996 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:35.074964 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:35.075896 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:35.075869 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2a57592f-d4b1-4ba5-959b-224b1557b3a7-hub\") pod \"cluster-proxy-proxy-agent-78d887b5fb-ggh9f\" (UID: \"2a57592f-d4b1-4ba5-959b-224b1557b3a7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d887b5fb-ggh9f" Aug 05 16:22:35.076015 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:35.075909 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a57592f-d4b1-4ba5-959b-224b1557b3a7-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78d887b5fb-ggh9f\" (UID: \"2a57592f-d4b1-4ba5-959b-224b1557b3a7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d887b5fb-ggh9f" Aug 05 16:22:35.076015 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:35.075965 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a57592f-d4b1-4ba5-959b-224b1557b3a7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78d887b5fb-ggh9f\" (UID: \"2a57592f-d4b1-4ba5-959b-224b1557b3a7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d887b5fb-ggh9f" Aug 05 16:22:35.076015 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:35.076006 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzg74\" (UniqueName: \"kubernetes.io/projected/2a57592f-d4b1-4ba5-959b-224b1557b3a7-kube-api-access-pzg74\") pod \"cluster-proxy-proxy-agent-78d887b5fb-ggh9f\" (UID: \"2a57592f-d4b1-4ba5-959b-224b1557b3a7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d887b5fb-ggh9f" Aug 05 16:22:35.076215 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:35.076039 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2a57592f-d4b1-4ba5-959b-224b1557b3a7-ca\") pod \"cluster-proxy-proxy-agent-78d887b5fb-ggh9f\" (UID: \"2a57592f-d4b1-4ba5-959b-224b1557b3a7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d887b5fb-ggh9f" Aug 05 16:22:35.076215 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:35.076067 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2a57592f-d4b1-4ba5-959b-224b1557b3a7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78d887b5fb-ggh9f\" (UID: \"2a57592f-d4b1-4ba5-959b-224b1557b3a7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d887b5fb-ggh9f" Aug 05 16:22:35.157433 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:35.157390 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:35.176723 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:35.176638 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2a57592f-d4b1-4ba5-959b-224b1557b3a7-hub\") pod \"cluster-proxy-proxy-agent-78d887b5fb-ggh9f\" (UID: \"2a57592f-d4b1-4ba5-959b-224b1557b3a7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d887b5fb-ggh9f" Aug 05 16:22:35.176723 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:35.176681 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a57592f-d4b1-4ba5-959b-224b1557b3a7-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78d887b5fb-ggh9f\" (UID: \"2a57592f-d4b1-4ba5-959b-224b1557b3a7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d887b5fb-ggh9f" Aug 05 16:22:35.176951 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:35.176823 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a57592f-d4b1-4ba5-959b-224b1557b3a7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78d887b5fb-ggh9f\" (UID: \"2a57592f-d4b1-4ba5-959b-224b1557b3a7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d887b5fb-ggh9f" Aug 05 16:22:35.176951 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:35.176884 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzg74\" (UniqueName: \"kubernetes.io/projected/2a57592f-d4b1-4ba5-959b-224b1557b3a7-kube-api-access-pzg74\") pod \"cluster-proxy-proxy-agent-78d887b5fb-ggh9f\" (UID: \"2a57592f-d4b1-4ba5-959b-224b1557b3a7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d887b5fb-ggh9f" Aug 05 16:22:35.176951 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:35.176927 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2a57592f-d4b1-4ba5-959b-224b1557b3a7-ca\") pod \"cluster-proxy-proxy-agent-78d887b5fb-ggh9f\" (UID: \"2a57592f-d4b1-4ba5-959b-224b1557b3a7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d887b5fb-ggh9f" Aug 05 16:22:35.177092 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:35.176956 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2a57592f-d4b1-4ba5-959b-224b1557b3a7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78d887b5fb-ggh9f\" (UID: \"2a57592f-d4b1-4ba5-959b-224b1557b3a7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d887b5fb-ggh9f" Aug 05 16:22:35.177506 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:35.177482 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a57592f-d4b1-4ba5-959b-224b1557b3a7-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78d887b5fb-ggh9f\" (UID: \"2a57592f-d4b1-4ba5-959b-224b1557b3a7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d887b5fb-ggh9f" Aug 05 16:22:35.179723 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:35.179695 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2a57592f-d4b1-4ba5-959b-224b1557b3a7-ca\") pod \"cluster-proxy-proxy-agent-78d887b5fb-ggh9f\" (UID: \"2a57592f-d4b1-4ba5-959b-224b1557b3a7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d887b5fb-ggh9f" Aug 05 16:22:35.179806 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:35.179726 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2a57592f-d4b1-4ba5-959b-224b1557b3a7-hub\") pod \"cluster-proxy-proxy-agent-78d887b5fb-ggh9f\" (UID: \"2a57592f-d4b1-4ba5-959b-224b1557b3a7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d887b5fb-ggh9f" Aug 05 16:22:35.179939 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:35.179919 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2a57592f-d4b1-4ba5-959b-224b1557b3a7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78d887b5fb-ggh9f\" (UID: \"2a57592f-d4b1-4ba5-959b-224b1557b3a7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d887b5fb-ggh9f" Aug 05 16:22:35.179975 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:35.179950 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a57592f-d4b1-4ba5-959b-224b1557b3a7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78d887b5fb-ggh9f\" (UID: \"2a57592f-d4b1-4ba5-959b-224b1557b3a7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d887b5fb-ggh9f" Aug 05 16:22:35.186341 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:35.186314 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzg74\" (UniqueName: \"kubernetes.io/projected/2a57592f-d4b1-4ba5-959b-224b1557b3a7-kube-api-access-pzg74\") pod \"cluster-proxy-proxy-agent-78d887b5fb-ggh9f\" (UID: \"2a57592f-d4b1-4ba5-959b-224b1557b3a7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d887b5fb-ggh9f" Aug 05 16:22:35.265108 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:35.265069 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:35.285420 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:35.285388 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d887b5fb-ggh9f" Aug 05 16:22:35.357651 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:35.357613 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:35.470738 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:35.470656 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:37.339092 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:37.339042 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d887b5fb-ggh9f"] Aug 05 16:22:37.342131 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:22:37.342096 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a57592f_d4b1_4ba5_959b_224b1557b3a7.slice/crio-d24f234c7118bc5be9b7906fc83f94fc93acb6400c6cd5b3a4e13c4a1af90aee WatchSource:0}: Error finding container d24f234c7118bc5be9b7906fc83f94fc93acb6400c6cd5b3a4e13c4a1af90aee: Status 404 returned error can't find the container with id d24f234c7118bc5be9b7906fc83f94fc93acb6400c6cd5b3a4e13c4a1af90aee Aug 05 16:22:37.597774 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:37.597743 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qjdbg" event={"ID":"39d233df-9262-45b2-8065-0d44a7ee9e12","Type":"ContainerStarted","Data":"9ee673e1d98386a4c3320ca7f304ffdff8ec22bd693717627e4f34f73236d9f8"} Aug 05 16:22:37.598880 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:37.598853 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d887b5fb-ggh9f" event={"ID":"2a57592f-d4b1-4ba5-959b-224b1557b3a7","Type":"ContainerStarted","Data":"d24f234c7118bc5be9b7906fc83f94fc93acb6400c6cd5b3a4e13c4a1af90aee"} Aug 05 16:22:37.600511 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:37.600483 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w8666" event={"ID":"89e8400b-bfeb-44c4-8302-59ab47d1b4cf","Type":"ContainerStarted","Data":"403eac94b1ba2813894adf10f7d2e1dd62aa9524b34e9ca96a8700a3f66e3fb3"} Aug 05 16:22:37.600621 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:37.600530 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w8666" event={"ID":"89e8400b-bfeb-44c4-8302-59ab47d1b4cf","Type":"ContainerStarted","Data":"0b969eddd29f590ff2ef87cec2291775bd02b294356221f2436e8603ca1bc201"} Aug 05 16:22:37.600621 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:37.600551 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w8666" Aug 05 16:22:37.601753 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:37.601734 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pwx27" event={"ID":"e303e40d-9669-4230-a1e1-c75ea3a2c4df","Type":"ContainerStarted","Data":"c91e9c6c12384b39811845885841f53cc1613412c3e0c386e3ccdac68fa33877"} Aug 05 16:22:37.601873 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:37.601860 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:22:37.637454 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:37.637406 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qjdbg" podStartSLOduration=66.918166752 podStartE2EDuration="1m12.637391296s" totalImagesPullingTime="5.719224544s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 16:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:22:37.618450384 +0000 UTC m=+72.775296286" watchObservedRunningTime="2026-08-05 16:22:37.637391296 +0000 UTC m=+72.794237187" Aug 05 16:22:37.637802 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:37.637781 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-w8666" podStartSLOduration=33.028659607 podStartE2EDuration="36.637776937s" totalImagesPullingTime="3.60911733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:22:37.636697604 +0000 UTC m=+72.793543504" watchObservedRunningTime="2026-08-05 16:22:37.637776937 +0000 UTC m=+72.794622837" Aug 05 16:22:37.654444 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:37.654403 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pwx27" podStartSLOduration=66.987934813 podStartE2EDuration="1m12.654371328s" totalImagesPullingTime="5.666436515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:22:37.65352565 +0000 UTC m=+72.810371530" watchObservedRunningTime="2026-08-05 16:22:37.654371328 +0000 UTC m=+72.811217301" Aug 05 16:22:39.462303 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:39.462265 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:39.533109 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:39.533073 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:39.865855 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:39.865813 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:40.818693 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:40.818649 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:40.888496 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:40.888451 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:42.572833 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:42.572797 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:42.625246 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:42.625216 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d887b5fb-ggh9f" event={"ID":"2a57592f-d4b1-4ba5-959b-224b1557b3a7","Type":"ContainerStarted","Data":"821796d4747e1c517beb5f96b40028300737bedd036808bb51fb205a1eeb77c7"} Aug 05 16:22:42.625395 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:42.625255 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d887b5fb-ggh9f" event={"ID":"2a57592f-d4b1-4ba5-959b-224b1557b3a7","Type":"ContainerStarted","Data":"2c2b5f2082bd4a6bd11b24e50ddfa8849957cdf4b8834af8a96b12df88a59067"} Aug 05 16:22:42.625395 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:42.625268 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d887b5fb-ggh9f" event={"ID":"2a57592f-d4b1-4ba5-959b-224b1557b3a7","Type":"ContainerStarted","Data":"a0a37008bbd0338b9a4a781acac7dbb621a46ff76c1ce5abd06da7c76ffd02c2"} Aug 05 16:22:42.646275 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:42.646221 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78d887b5fb-ggh9f" podStartSLOduration=4.143733959 podStartE2EDuration="8.64620626s" totalImagesPullingTime="4.502472301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:22:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:22:42.645897461 +0000 UTC m=+77.802743361" watchObservedRunningTime="2026-08-05 16:22:42.64620626 +0000 UTC m=+77.803052160" Aug 05 16:22:44.630767 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:44.630723 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:44.829404 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:44.829335 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:44.955909 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:44.955810 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ffzbh"] Aug 05 16:22:44.960665 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:44.960636 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:44.965594 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:44.965573 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 05 16:22:44.966845 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:44.966827 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 05 16:22:44.966942 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:44.966855 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zxk8h\"" Aug 05 16:22:44.966942 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:44.966876 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 05 16:22:44.967059 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:44.967025 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 05 16:22:44.967120 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:44.967059 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 05 16:22:44.967892 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:44.967874 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 05 16:22:44.975118 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:44.975085 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:45.057109 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.057075 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/44168e30-8669-4130-bfe0-f64e6938aae4-node-exporter-textfile\") pod \"node-exporter-ffzbh\" (UID: \"44168e30-8669-4130-bfe0-f64e6938aae4\") " pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:45.057109 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.057113 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/44168e30-8669-4130-bfe0-f64e6938aae4-root\") pod \"node-exporter-ffzbh\" (UID: \"44168e30-8669-4130-bfe0-f64e6938aae4\") " pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:45.057367 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.057141 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzjtc\" (UniqueName: \"kubernetes.io/projected/44168e30-8669-4130-bfe0-f64e6938aae4-kube-api-access-tzjtc\") pod \"node-exporter-ffzbh\" (UID: \"44168e30-8669-4130-bfe0-f64e6938aae4\") " pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:45.057367 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.057218 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/44168e30-8669-4130-bfe0-f64e6938aae4-node-exporter-accelerators-collector-config\") pod \"node-exporter-ffzbh\" (UID: \"44168e30-8669-4130-bfe0-f64e6938aae4\") " pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:45.057367 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.057262 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/44168e30-8669-4130-bfe0-f64e6938aae4-node-exporter-tls\") pod \"node-exporter-ffzbh\" (UID: \"44168e30-8669-4130-bfe0-f64e6938aae4\") " pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:45.057367 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.057303 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/44168e30-8669-4130-bfe0-f64e6938aae4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ffzbh\" (UID: \"44168e30-8669-4130-bfe0-f64e6938aae4\") " pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:45.057367 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.057349 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44168e30-8669-4130-bfe0-f64e6938aae4-sys\") pod \"node-exporter-ffzbh\" (UID: \"44168e30-8669-4130-bfe0-f64e6938aae4\") " pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:45.057600 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.057399 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44168e30-8669-4130-bfe0-f64e6938aae4-metrics-client-ca\") pod \"node-exporter-ffzbh\" (UID: \"44168e30-8669-4130-bfe0-f64e6938aae4\") " pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:45.057600 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.057431 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/44168e30-8669-4130-bfe0-f64e6938aae4-node-exporter-wtmp\") pod \"node-exporter-ffzbh\" (UID: \"44168e30-8669-4130-bfe0-f64e6938aae4\") " pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:45.124591 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.124545 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:45.157948 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.157920 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/44168e30-8669-4130-bfe0-f64e6938aae4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ffzbh\" (UID: \"44168e30-8669-4130-bfe0-f64e6938aae4\") " pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:45.158038 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.157960 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44168e30-8669-4130-bfe0-f64e6938aae4-sys\") pod \"node-exporter-ffzbh\" (UID: \"44168e30-8669-4130-bfe0-f64e6938aae4\") " pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:45.158038 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.157982 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44168e30-8669-4130-bfe0-f64e6938aae4-metrics-client-ca\") pod \"node-exporter-ffzbh\" (UID: \"44168e30-8669-4130-bfe0-f64e6938aae4\") " pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:45.158038 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.158001 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/44168e30-8669-4130-bfe0-f64e6938aae4-node-exporter-wtmp\") pod \"node-exporter-ffzbh\" (UID: \"44168e30-8669-4130-bfe0-f64e6938aae4\") " pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:45.158189 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.158116 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/44168e30-8669-4130-bfe0-f64e6938aae4-node-exporter-wtmp\") pod \"node-exporter-ffzbh\" (UID: \"44168e30-8669-4130-bfe0-f64e6938aae4\") " pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:45.158189 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.158128 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44168e30-8669-4130-bfe0-f64e6938aae4-sys\") pod \"node-exporter-ffzbh\" (UID: \"44168e30-8669-4130-bfe0-f64e6938aae4\") " pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:45.158189 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.158144 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/44168e30-8669-4130-bfe0-f64e6938aae4-node-exporter-textfile\") pod \"node-exporter-ffzbh\" (UID: \"44168e30-8669-4130-bfe0-f64e6938aae4\") " pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:45.158189 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.158179 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/44168e30-8669-4130-bfe0-f64e6938aae4-root\") pod \"node-exporter-ffzbh\" (UID: \"44168e30-8669-4130-bfe0-f64e6938aae4\") " pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:45.158397 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.158201 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzjtc\" (UniqueName: \"kubernetes.io/projected/44168e30-8669-4130-bfe0-f64e6938aae4-kube-api-access-tzjtc\") pod \"node-exporter-ffzbh\" (UID: \"44168e30-8669-4130-bfe0-f64e6938aae4\") " pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:45.158397 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.158222 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/44168e30-8669-4130-bfe0-f64e6938aae4-node-exporter-accelerators-collector-config\") pod \"node-exporter-ffzbh\" (UID: \"44168e30-8669-4130-bfe0-f64e6938aae4\") " pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:45.158397 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.158240 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/44168e30-8669-4130-bfe0-f64e6938aae4-node-exporter-tls\") pod \"node-exporter-ffzbh\" (UID: \"44168e30-8669-4130-bfe0-f64e6938aae4\") " pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:45.158397 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.158316 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/44168e30-8669-4130-bfe0-f64e6938aae4-root\") pod \"node-exporter-ffzbh\" (UID: \"44168e30-8669-4130-bfe0-f64e6938aae4\") " pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:45.158588 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.158482 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/44168e30-8669-4130-bfe0-f64e6938aae4-node-exporter-textfile\") pod \"node-exporter-ffzbh\" (UID: \"44168e30-8669-4130-bfe0-f64e6938aae4\") " pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:45.158751 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.158729 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44168e30-8669-4130-bfe0-f64e6938aae4-metrics-client-ca\") pod \"node-exporter-ffzbh\" (UID: \"44168e30-8669-4130-bfe0-f64e6938aae4\") " pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:45.158823 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.158772 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/44168e30-8669-4130-bfe0-f64e6938aae4-node-exporter-accelerators-collector-config\") pod \"node-exporter-ffzbh\" (UID: \"44168e30-8669-4130-bfe0-f64e6938aae4\") " pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:45.160349 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.160322 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/44168e30-8669-4130-bfe0-f64e6938aae4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ffzbh\" (UID: \"44168e30-8669-4130-bfe0-f64e6938aae4\") " pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:45.160567 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.160549 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/44168e30-8669-4130-bfe0-f64e6938aae4-node-exporter-tls\") pod \"node-exporter-ffzbh\" (UID: \"44168e30-8669-4130-bfe0-f64e6938aae4\") " pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:45.173221 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.173195 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzjtc\" (UniqueName: \"kubernetes.io/projected/44168e30-8669-4130-bfe0-f64e6938aae4-kube-api-access-tzjtc\") pod \"node-exporter-ffzbh\" (UID: \"44168e30-8669-4130-bfe0-f64e6938aae4\") " pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:45.221185 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.221108 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:45.272326 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.272296 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ffzbh" Aug 05 16:22:45.284450 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:22:45.284421 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44168e30_8669_4130_bfe0_f64e6938aae4.slice/crio-bcc5149e553a098844d003b70735109c66e66610bf762517a67db3faa2731904 WatchSource:0}: Error finding container bcc5149e553a098844d003b70735109c66e66610bf762517a67db3faa2731904: Status 404 returned error can't find the container with id bcc5149e553a098844d003b70735109c66e66610bf762517a67db3faa2731904 Aug 05 16:22:45.324136 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.324077 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:45.389894 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.389858 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:45.454798 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.454769 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:45.636819 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.636785 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ffzbh" event={"ID":"44168e30-8669-4130-bfe0-f64e6938aae4","Type":"ContainerStarted","Data":"bcc5149e553a098844d003b70735109c66e66610bf762517a67db3faa2731904"} Aug 05 16:22:45.808185 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.808150 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:45.933601 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:45.933523 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:46.366111 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:46.366082 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:46.612703 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:46.612669 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:46.712810 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:46.712734 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:47.284773 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:47.284742 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:47.384341 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:47.384313 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:47.609152 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:47.609076 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w8666" Aug 05 16:22:47.644884 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:47.644855 2573 generic.go:356] "Generic (PLEG): container finished" podID="44168e30-8669-4130-bfe0-f64e6938aae4" containerID="f4e8d7b3aa2a5dce15a80d6a105cc1a30a682382c44edf6b86eb4c9ac11a77c5" exitCode=0 Aug 05 16:22:47.645043 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:47.644943 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ffzbh" event={"ID":"44168e30-8669-4130-bfe0-f64e6938aae4","Type":"ContainerDied","Data":"f4e8d7b3aa2a5dce15a80d6a105cc1a30a682382c44edf6b86eb4c9ac11a77c5"} Aug 05 16:22:47.925787 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:47.925694 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:48.121103 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:48.121062 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:48.215955 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:48.215841 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:48.453076 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:48.453043 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:48.650360 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:48.650318 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ffzbh" event={"ID":"44168e30-8669-4130-bfe0-f64e6938aae4","Type":"ContainerStarted","Data":"d3e2c35fd542c510c72947f22815d7fa28911e9351da9cd7c9bfb110a7342a17"} Aug 05 16:22:48.650573 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:48.650436 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ffzbh" event={"ID":"44168e30-8669-4130-bfe0-f64e6938aae4","Type":"ContainerStarted","Data":"c85abacbb3dcf08671aaf9e9dabb06d9622fcfba197dd651f99524510b8abfef"} Aug 05 16:22:48.681572 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:48.681528 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ffzbh" podStartSLOduration=2.950926456 podStartE2EDuration="4.681505443s" totalImagesPullingTime="1.669680973s" totalInitContainerRuntime="60.898014ms" isStatefulPod=true podCreationTimestamp="2026-08-05 16:22:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:22:48.67966064 +0000 UTC m=+83.836506540" watchObservedRunningTime="2026-08-05 16:22:48.681505443 +0000 UTC m=+83.838351345" Aug 05 16:22:49.627319 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:49.627275 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:49.723570 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:49.723537 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:49.826996 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:49.826949 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:49.928095 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:49.928017 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:50.031508 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:50.031470 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:50.229568 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:50.229464 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:50.355668 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:50.355638 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:51.526742 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:51.526708 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:53.181989 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:53.181958 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:53.322756 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:53.322721 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:56.374744 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:56.374712 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:56.479251 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:56.479217 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:56.812975 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:56.812945 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:56.914893 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:56.914844 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:57.326906 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:57.326878 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:57.410150 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:57.410120 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:58.396037 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:58.395996 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:58.516267 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:58.516201 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:58.839327 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:58.839284 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:58.939071 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:58.939043 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:59.008069 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:59.008035 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:59.187277 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:59.187164 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:22:59.287255 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:22:59.287220 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:00.981824 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:23:00.981793 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:01.087437 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:23:01.087408 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:01.179695 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:23:01.179669 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:08.610416 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:23:08.610387 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pwx27" Aug 05 16:23:12.317744 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:23:12.317697 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:12.377641 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:23:12.377608 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:21.951727 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:23:21.951692 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:27.306851 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:23:27.306818 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:27.372012 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:23:27.371972 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:33.572721 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:23:33.572687 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:23:44.837609 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:23:44.837571 2573 generic.go:356] "Generic (PLEG): container finished" podID="791538ee-cf4d-4198-aed7-9ac30d92460e" containerID="32608dbde2c0ca3fb7f6980d9e7f4dd332401684212e83efb246be68c59ab581" exitCode=0 Aug 05 16:23:44.838137 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:23:44.837650 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-stpfk" event={"ID":"791538ee-cf4d-4198-aed7-9ac30d92460e","Type":"ContainerDied","Data":"32608dbde2c0ca3fb7f6980d9e7f4dd332401684212e83efb246be68c59ab581"} Aug 05 16:23:44.838137 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:23:44.838073 2573 scope.go:117] "RemoveContainer" containerID="32608dbde2c0ca3fb7f6980d9e7f4dd332401684212e83efb246be68c59ab581" Aug 05 16:23:45.843410 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:23:45.843358 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-stpfk" event={"ID":"791538ee-cf4d-4198-aed7-9ac30d92460e","Type":"ContainerStarted","Data":"22c95b89073a96cb9ffa6e79958177b5dbe82f3b84c26f10dd4c332fdf9f124e"} Aug 05 16:24:04.491996 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:24:04.491917 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:04.677293 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:24:04.677262 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:13.374097 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:24:13.374063 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:13.566287 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:24:13.566257 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:13.666267 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:24:13.666178 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:14.093758 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:24:14.093725 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:14.154038 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:24:14.154001 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:24.269429 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:24:24.269335 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:24.366295 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:24:24.366257 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:24:24.469244 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:24:24.469207 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:25:11.810146 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:25:11.810112 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:25:11.876754 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:25:11.876729 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:25:19.734919 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:25:19.734880 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:25:19.874884 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:25:19.874844 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:25:21.222531 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:25:21.222495 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:25:21.464512 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:25:21.464483 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:25:31.650049 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:25:31.650007 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:25:31.738579 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:25:31.738548 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:25:32.552019 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:25:32.551989 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:25:32.652307 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:25:32.652277 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:25:32.720183 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:25:32.720155 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:25:42.319913 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:25:42.319880 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:25:42.420867 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:25:42.420812 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:25:54.213078 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:25:54.213005 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:26:14.903370 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:26:14.903337 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:26:25.214760 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:26:25.214724 2573 kubelet.go:1648] "Image garbage collection succeeded" Aug 05 16:27:05.561244 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:05.561212 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:06.869565 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:06.869530 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:27.651407 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:27.651309 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:28.990263 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:28.990227 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:31.507067 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:31.507033 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:31.611010 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:31.610945 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:33.122997 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:33.122968 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:33.222205 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:33.222174 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:33.322918 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:33.322886 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:35.522090 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:35.522039 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:35.715914 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:35.715882 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:37.019167 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:37.019135 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:37.120121 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:37.120090 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:38.210052 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:38.210015 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:38.312451 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:38.312419 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:41.918003 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:41.917967 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:42.018484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:42.018451 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:42.704728 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:42.704696 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:43.543520 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:43.543489 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-qm4nw"] Aug 05 16:27:43.545530 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:43.545514 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-qm4nw" Aug 05 16:27:43.548565 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:43.548537 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Aug 05 16:27:43.548687 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:43.548651 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Aug 05 16:27:43.549937 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:43.549922 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-jss22\"" Aug 05 16:27:43.558835 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:43.558814 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-qm4nw"] Aug 05 16:27:43.651486 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:43.651455 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce22f555-f9bc-4fe3-9693-05b9251d7fe9-bound-sa-token\") pod \"cert-manager-55b4d954d5-qm4nw\" (UID: \"ce22f555-f9bc-4fe3-9693-05b9251d7fe9\") " pod="cert-manager/cert-manager-55b4d954d5-qm4nw" Aug 05 16:27:43.651643 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:43.651517 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b4l6\" (UniqueName: \"kubernetes.io/projected/ce22f555-f9bc-4fe3-9693-05b9251d7fe9-kube-api-access-9b4l6\") pod \"cert-manager-55b4d954d5-qm4nw\" (UID: \"ce22f555-f9bc-4fe3-9693-05b9251d7fe9\") " pod="cert-manager/cert-manager-55b4d954d5-qm4nw" Aug 05 16:27:43.752099 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:43.752066 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b4l6\" (UniqueName: \"kubernetes.io/projected/ce22f555-f9bc-4fe3-9693-05b9251d7fe9-kube-api-access-9b4l6\") pod \"cert-manager-55b4d954d5-qm4nw\" (UID: \"ce22f555-f9bc-4fe3-9693-05b9251d7fe9\") " pod="cert-manager/cert-manager-55b4d954d5-qm4nw" Aug 05 16:27:43.752252 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:43.752108 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce22f555-f9bc-4fe3-9693-05b9251d7fe9-bound-sa-token\") pod \"cert-manager-55b4d954d5-qm4nw\" (UID: \"ce22f555-f9bc-4fe3-9693-05b9251d7fe9\") " pod="cert-manager/cert-manager-55b4d954d5-qm4nw" Aug 05 16:27:43.763867 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:43.763833 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce22f555-f9bc-4fe3-9693-05b9251d7fe9-bound-sa-token\") pod \"cert-manager-55b4d954d5-qm4nw\" (UID: \"ce22f555-f9bc-4fe3-9693-05b9251d7fe9\") " pod="cert-manager/cert-manager-55b4d954d5-qm4nw" Aug 05 16:27:43.764489 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:43.764473 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b4l6\" (UniqueName: \"kubernetes.io/projected/ce22f555-f9bc-4fe3-9693-05b9251d7fe9-kube-api-access-9b4l6\") pod \"cert-manager-55b4d954d5-qm4nw\" (UID: \"ce22f555-f9bc-4fe3-9693-05b9251d7fe9\") " pod="cert-manager/cert-manager-55b4d954d5-qm4nw" Aug 05 16:27:43.853990 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:43.853904 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-qm4nw" Aug 05 16:27:43.977659 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:43.977628 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-qm4nw"] Aug 05 16:27:43.980488 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:27:43.980455 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce22f555_f9bc_4fe3_9693_05b9251d7fe9.slice/crio-c617f9d701acf13518e049ae0b959db31a6df425c82867fd7e2c49fc7cd8790c WatchSource:0}: Error finding container c617f9d701acf13518e049ae0b959db31a6df425c82867fd7e2c49fc7cd8790c: Status 404 returned error can't find the container with id c617f9d701acf13518e049ae0b959db31a6df425c82867fd7e2c49fc7cd8790c Aug 05 16:27:43.982365 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:43.982349 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 05 16:27:44.617931 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:44.617869 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:44.642120 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:44.642078 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-qm4nw" event={"ID":"ce22f555-f9bc-4fe3-9693-05b9251d7fe9","Type":"ContainerStarted","Data":"c617f9d701acf13518e049ae0b959db31a6df425c82867fd7e2c49fc7cd8790c"} Aug 05 16:27:44.717419 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:44.717385 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:44.972586 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:44.972493 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:45.038760 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:45.038695 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:45.107370 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:45.107332 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:45.691611 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:45.691566 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:45.791344 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:45.791309 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:47.654782 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:47.654742 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-qm4nw" event={"ID":"ce22f555-f9bc-4fe3-9693-05b9251d7fe9","Type":"ContainerStarted","Data":"b84cb9c9b80be0b4e612fe86b6da28177a650dc99907afeab34fbddc1425d870"} Aug 05 16:27:47.673883 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:47.673837 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-qm4nw" podStartSLOduration=1.817524911 podStartE2EDuration="4.673822584s" totalImagesPullingTime="2.856297673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:27:47.672919905 +0000 UTC m=+382.829765817" watchObservedRunningTime="2026-08-05 16:27:47.673822584 +0000 UTC m=+382.830668485" Aug 05 16:27:50.546963 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:50.546931 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:50.649026 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:50.648989 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:52.563809 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:52.563776 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:53.804192 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:53.804159 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:53.919039 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:53.919011 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:55.740248 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:55.740218 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:55.842173 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:55.842138 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:56.152603 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:56.152572 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:56.252631 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:56.252602 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:27:56.323899 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:27:56.323871 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:05.048651 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:05.048616 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:05.172086 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:05.172054 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:05.681809 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:05.681771 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:06.191618 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.191581 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:06.299495 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.299463 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:06.338987 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.338958 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9778877d8-chcjf"] Aug 05 16:28:06.343187 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.343168 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9778877d8-chcjf" Aug 05 16:28:06.347323 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.347296 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 05 16:28:06.347450 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.347390 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 05 16:28:06.348265 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.348247 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 05 16:28:06.348421 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.348405 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 05 16:28:06.353318 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.353303 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jcslj\"" Aug 05 16:28:06.362170 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.362150 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9778877d8-chcjf"] Aug 05 16:28:06.420415 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.420366 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f2370e77-a0e4-4344-8ae8-eb4d6ed2f987-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9778877d8-chcjf\" (UID: \"f2370e77-a0e4-4344-8ae8-eb4d6ed2f987\") " pod="opendatahub/opendatahub-operator-controller-manager-9778877d8-chcjf" Aug 05 16:28:06.420415 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.420420 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2370e77-a0e4-4344-8ae8-eb4d6ed2f987-webhook-cert\") pod \"opendatahub-operator-controller-manager-9778877d8-chcjf\" (UID: \"f2370e77-a0e4-4344-8ae8-eb4d6ed2f987\") " pod="opendatahub/opendatahub-operator-controller-manager-9778877d8-chcjf" Aug 05 16:28:06.420658 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.420509 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2370e77-a0e4-4344-8ae8-eb4d6ed2f987-tmp\") pod \"opendatahub-operator-controller-manager-9778877d8-chcjf\" (UID: \"f2370e77-a0e4-4344-8ae8-eb4d6ed2f987\") " pod="opendatahub/opendatahub-operator-controller-manager-9778877d8-chcjf" Aug 05 16:28:06.420658 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.420543 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rvms\" (UniqueName: \"kubernetes.io/projected/f2370e77-a0e4-4344-8ae8-eb4d6ed2f987-kube-api-access-5rvms\") pod \"opendatahub-operator-controller-manager-9778877d8-chcjf\" (UID: \"f2370e77-a0e4-4344-8ae8-eb4d6ed2f987\") " pod="opendatahub/opendatahub-operator-controller-manager-9778877d8-chcjf" Aug 05 16:28:06.420658 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.420573 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f2370e77-a0e4-4344-8ae8-eb4d6ed2f987-manifests\") pod \"opendatahub-operator-controller-manager-9778877d8-chcjf\" (UID: \"f2370e77-a0e4-4344-8ae8-eb4d6ed2f987\") " pod="opendatahub/opendatahub-operator-controller-manager-9778877d8-chcjf" Aug 05 16:28:06.521494 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.521466 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f2370e77-a0e4-4344-8ae8-eb4d6ed2f987-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9778877d8-chcjf\" (UID: \"f2370e77-a0e4-4344-8ae8-eb4d6ed2f987\") " pod="opendatahub/opendatahub-operator-controller-manager-9778877d8-chcjf" Aug 05 16:28:06.521657 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.521511 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2370e77-a0e4-4344-8ae8-eb4d6ed2f987-webhook-cert\") pod \"opendatahub-operator-controller-manager-9778877d8-chcjf\" (UID: \"f2370e77-a0e4-4344-8ae8-eb4d6ed2f987\") " pod="opendatahub/opendatahub-operator-controller-manager-9778877d8-chcjf" Aug 05 16:28:06.521657 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.521547 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2370e77-a0e4-4344-8ae8-eb4d6ed2f987-tmp\") pod \"opendatahub-operator-controller-manager-9778877d8-chcjf\" (UID: \"f2370e77-a0e4-4344-8ae8-eb4d6ed2f987\") " pod="opendatahub/opendatahub-operator-controller-manager-9778877d8-chcjf" Aug 05 16:28:06.521657 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.521566 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rvms\" (UniqueName: \"kubernetes.io/projected/f2370e77-a0e4-4344-8ae8-eb4d6ed2f987-kube-api-access-5rvms\") pod \"opendatahub-operator-controller-manager-9778877d8-chcjf\" (UID: \"f2370e77-a0e4-4344-8ae8-eb4d6ed2f987\") " pod="opendatahub/opendatahub-operator-controller-manager-9778877d8-chcjf" Aug 05 16:28:06.521657 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.521583 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f2370e77-a0e4-4344-8ae8-eb4d6ed2f987-manifests\") pod \"opendatahub-operator-controller-manager-9778877d8-chcjf\" (UID: \"f2370e77-a0e4-4344-8ae8-eb4d6ed2f987\") " pod="opendatahub/opendatahub-operator-controller-manager-9778877d8-chcjf" Aug 05 16:28:06.521993 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.521968 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f2370e77-a0e4-4344-8ae8-eb4d6ed2f987-manifests\") pod \"opendatahub-operator-controller-manager-9778877d8-chcjf\" (UID: \"f2370e77-a0e4-4344-8ae8-eb4d6ed2f987\") " pod="opendatahub/opendatahub-operator-controller-manager-9778877d8-chcjf" Aug 05 16:28:06.524080 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.524052 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f2370e77-a0e4-4344-8ae8-eb4d6ed2f987-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9778877d8-chcjf\" (UID: \"f2370e77-a0e4-4344-8ae8-eb4d6ed2f987\") " pod="opendatahub/opendatahub-operator-controller-manager-9778877d8-chcjf" Aug 05 16:28:06.524205 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.524088 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2370e77-a0e4-4344-8ae8-eb4d6ed2f987-webhook-cert\") pod \"opendatahub-operator-controller-manager-9778877d8-chcjf\" (UID: \"f2370e77-a0e4-4344-8ae8-eb4d6ed2f987\") " pod="opendatahub/opendatahub-operator-controller-manager-9778877d8-chcjf" Aug 05 16:28:06.524272 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.524235 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2370e77-a0e4-4344-8ae8-eb4d6ed2f987-tmp\") pod \"opendatahub-operator-controller-manager-9778877d8-chcjf\" (UID: \"f2370e77-a0e4-4344-8ae8-eb4d6ed2f987\") " pod="opendatahub/opendatahub-operator-controller-manager-9778877d8-chcjf" Aug 05 16:28:06.541388 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.541356 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rvms\" (UniqueName: \"kubernetes.io/projected/f2370e77-a0e4-4344-8ae8-eb4d6ed2f987-kube-api-access-5rvms\") pod \"opendatahub-operator-controller-manager-9778877d8-chcjf\" (UID: \"f2370e77-a0e4-4344-8ae8-eb4d6ed2f987\") " pod="opendatahub/opendatahub-operator-controller-manager-9778877d8-chcjf" Aug 05 16:28:06.653829 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.653792 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9778877d8-chcjf" Aug 05 16:28:06.705245 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.705207 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:06.798187 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.798112 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9778877d8-chcjf"] Aug 05 16:28:06.802066 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:28:06.802033 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2370e77_a0e4_4344_8ae8_eb4d6ed2f987.slice/crio-7543ed9c56beb3f6c73b10cefa6715e2b2ae73b070abf344951c43d4d9b05b4c WatchSource:0}: Error finding container 7543ed9c56beb3f6c73b10cefa6715e2b2ae73b070abf344951c43d4d9b05b4c: Status 404 returned error can't find the container with id 7543ed9c56beb3f6c73b10cefa6715e2b2ae73b070abf344951c43d4d9b05b4c Aug 05 16:28:06.839166 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.839139 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:06.957602 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:06.957569 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:07.057663 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:07.057585 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:07.203040 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:07.203004 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:07.725446 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:07.725403 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9778877d8-chcjf" event={"ID":"f2370e77-a0e4-4344-8ae8-eb4d6ed2f987","Type":"ContainerStarted","Data":"7543ed9c56beb3f6c73b10cefa6715e2b2ae73b070abf344951c43d4d9b05b4c"} Aug 05 16:28:10.739040 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:10.739008 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9778877d8-chcjf" event={"ID":"f2370e77-a0e4-4344-8ae8-eb4d6ed2f987","Type":"ContainerStarted","Data":"e87a9d24ec4464fd74b432a3090b7cfe8f0246f9d11e426b35007b18ed3a3f52"} Aug 05 16:28:11.744453 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:11.744417 2573 generic.go:356] "Generic (PLEG): container finished" podID="f2370e77-a0e4-4344-8ae8-eb4d6ed2f987" containerID="e87a9d24ec4464fd74b432a3090b7cfe8f0246f9d11e426b35007b18ed3a3f52" exitCode=0 Aug 05 16:28:11.744453 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:11.744457 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9778877d8-chcjf" event={"ID":"f2370e77-a0e4-4344-8ae8-eb4d6ed2f987","Type":"ContainerDied","Data":"e87a9d24ec4464fd74b432a3090b7cfe8f0246f9d11e426b35007b18ed3a3f52"} Aug 05 16:28:12.750942 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:12.750902 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9778877d8-chcjf" event={"ID":"f2370e77-a0e4-4344-8ae8-eb4d6ed2f987","Type":"ContainerStarted","Data":"0ab70b2c484439b9b988f3dfee3d946fd19ed900550b8ddcecb1c5ba3842376a"} Aug 05 16:28:12.751363 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:12.751017 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9778877d8-chcjf" Aug 05 16:28:12.777899 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:12.777848 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9778877d8-chcjf" podStartSLOduration=2.340180902 podStartE2EDuration="6.777828454s" totalImagesPullingTime="3.394320844s" totalInitContainerRuntime="1.043326708s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:28:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:28:12.775722686 +0000 UTC m=+407.932568598" watchObservedRunningTime="2026-08-05 16:28:12.777828454 +0000 UTC m=+407.934674354" Aug 05 16:28:14.864080 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:14.864049 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:14.971227 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:14.971197 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:17.450419 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:17.450368 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:17.586833 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:17.586795 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:23.757739 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:23.757710 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9778877d8-chcjf" Aug 05 16:28:25.504507 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:25.504472 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6677bbd944-snbv2"] Aug 05 16:28:25.508791 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:25.508767 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6677bbd944-snbv2" Aug 05 16:28:25.512941 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:25.512914 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Aug 05 16:28:25.513075 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:25.513060 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Aug 05 16:28:25.513134 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:25.513095 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xdl5l\"" Aug 05 16:28:25.513188 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:25.513154 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Aug 05 16:28:25.513353 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:25.513335 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Aug 05 16:28:25.513980 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:25.513960 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Aug 05 16:28:25.518477 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:25.518448 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6677bbd944-snbv2"] Aug 05 16:28:25.543538 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:25.543508 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:25.645112 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:25.645075 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:25.672279 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:25.672243 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0731c408-661d-4be4-8388-e1fc1cb301ab-manager-config\") pod \"lws-controller-manager-6677bbd944-snbv2\" (UID: \"0731c408-661d-4be4-8388-e1fc1cb301ab\") " pod="openshift-lws-operator/lws-controller-manager-6677bbd944-snbv2" Aug 05 16:28:25.672451 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:25.672289 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ntjf\" (UniqueName: \"kubernetes.io/projected/0731c408-661d-4be4-8388-e1fc1cb301ab-kube-api-access-2ntjf\") pod \"lws-controller-manager-6677bbd944-snbv2\" (UID: \"0731c408-661d-4be4-8388-e1fc1cb301ab\") " pod="openshift-lws-operator/lws-controller-manager-6677bbd944-snbv2" Aug 05 16:28:25.672451 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:25.672320 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0731c408-661d-4be4-8388-e1fc1cb301ab-cert\") pod \"lws-controller-manager-6677bbd944-snbv2\" (UID: \"0731c408-661d-4be4-8388-e1fc1cb301ab\") " pod="openshift-lws-operator/lws-controller-manager-6677bbd944-snbv2" Aug 05 16:28:25.672451 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:25.672336 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0731c408-661d-4be4-8388-e1fc1cb301ab-metrics-cert\") pod \"lws-controller-manager-6677bbd944-snbv2\" (UID: \"0731c408-661d-4be4-8388-e1fc1cb301ab\") " pod="openshift-lws-operator/lws-controller-manager-6677bbd944-snbv2" Aug 05 16:28:25.772932 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:25.772895 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ntjf\" (UniqueName: \"kubernetes.io/projected/0731c408-661d-4be4-8388-e1fc1cb301ab-kube-api-access-2ntjf\") pod \"lws-controller-manager-6677bbd944-snbv2\" (UID: \"0731c408-661d-4be4-8388-e1fc1cb301ab\") " pod="openshift-lws-operator/lws-controller-manager-6677bbd944-snbv2" Aug 05 16:28:25.773104 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:25.772944 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0731c408-661d-4be4-8388-e1fc1cb301ab-cert\") pod \"lws-controller-manager-6677bbd944-snbv2\" (UID: \"0731c408-661d-4be4-8388-e1fc1cb301ab\") " pod="openshift-lws-operator/lws-controller-manager-6677bbd944-snbv2" Aug 05 16:28:25.773104 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:25.772960 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0731c408-661d-4be4-8388-e1fc1cb301ab-metrics-cert\") pod \"lws-controller-manager-6677bbd944-snbv2\" (UID: \"0731c408-661d-4be4-8388-e1fc1cb301ab\") " pod="openshift-lws-operator/lws-controller-manager-6677bbd944-snbv2" Aug 05 16:28:25.773104 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:25.773013 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0731c408-661d-4be4-8388-e1fc1cb301ab-manager-config\") pod \"lws-controller-manager-6677bbd944-snbv2\" (UID: \"0731c408-661d-4be4-8388-e1fc1cb301ab\") " pod="openshift-lws-operator/lws-controller-manager-6677bbd944-snbv2" Aug 05 16:28:25.773779 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:25.773760 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0731c408-661d-4be4-8388-e1fc1cb301ab-manager-config\") pod \"lws-controller-manager-6677bbd944-snbv2\" (UID: \"0731c408-661d-4be4-8388-e1fc1cb301ab\") " pod="openshift-lws-operator/lws-controller-manager-6677bbd944-snbv2" Aug 05 16:28:25.775391 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:25.775347 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0731c408-661d-4be4-8388-e1fc1cb301ab-metrics-cert\") pod \"lws-controller-manager-6677bbd944-snbv2\" (UID: \"0731c408-661d-4be4-8388-e1fc1cb301ab\") " pod="openshift-lws-operator/lws-controller-manager-6677bbd944-snbv2" Aug 05 16:28:25.775516 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:25.775495 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0731c408-661d-4be4-8388-e1fc1cb301ab-cert\") pod \"lws-controller-manager-6677bbd944-snbv2\" (UID: \"0731c408-661d-4be4-8388-e1fc1cb301ab\") " pod="openshift-lws-operator/lws-controller-manager-6677bbd944-snbv2" Aug 05 16:28:25.794256 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:25.794231 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ntjf\" (UniqueName: \"kubernetes.io/projected/0731c408-661d-4be4-8388-e1fc1cb301ab-kube-api-access-2ntjf\") pod \"lws-controller-manager-6677bbd944-snbv2\" (UID: \"0731c408-661d-4be4-8388-e1fc1cb301ab\") " pod="openshift-lws-operator/lws-controller-manager-6677bbd944-snbv2" Aug 05 16:28:25.819070 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:25.819035 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6677bbd944-snbv2" Aug 05 16:28:25.967800 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:25.967771 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6677bbd944-snbv2"] Aug 05 16:28:25.969765 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:28:25.969738 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0731c408_661d_4be4_8388_e1fc1cb301ab.slice/crio-c08de6001c704484a789c4af0b169c0db6a5403ed6af4aeba97a6c02f87c8040 WatchSource:0}: Error finding container c08de6001c704484a789c4af0b169c0db6a5403ed6af4aeba97a6c02f87c8040: Status 404 returned error can't find the container with id c08de6001c704484a789c4af0b169c0db6a5403ed6af4aeba97a6c02f87c8040 Aug 05 16:28:26.495963 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:26.495933 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:26.636114 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:26.636082 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:26.771635 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:26.771535 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:26.801486 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:26.801232 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6677bbd944-snbv2" event={"ID":"0731c408-661d-4be4-8388-e1fc1cb301ab","Type":"ContainerStarted","Data":"c08de6001c704484a789c4af0b169c0db6a5403ed6af4aeba97a6c02f87c8040"} Aug 05 16:28:26.949705 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:26.949609 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:27.496834 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:27.496801 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:27.757031 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:27.756954 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:28.809521 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:28.809486 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6677bbd944-snbv2" event={"ID":"0731c408-661d-4be4-8388-e1fc1cb301ab","Type":"ContainerStarted","Data":"5f52dad0602e50b931aedfb963ce7d43df686233817ab49c14a36a1ea27c696a"} Aug 05 16:28:28.809892 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:28.809599 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6677bbd944-snbv2" Aug 05 16:28:28.842437 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:28.842370 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6677bbd944-snbv2" podStartSLOduration=1.448347147 podStartE2EDuration="3.842355139s" totalImagesPullingTime="2.394007992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:28:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:28:28.84054719 +0000 UTC m=+423.997393090" watchObservedRunningTime="2026-08-05 16:28:28.842355139 +0000 UTC m=+423.999201040" Aug 05 16:28:36.768494 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:36.768453 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:37.123215 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:37.123138 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:37.354095 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:37.354051 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:37.504309 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:37.504231 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:37.604095 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:37.604058 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:37.720318 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:37.720286 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:37.834542 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:37.834511 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:38.001136 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:37.993964 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:39.107772 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:39.107733 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:39.195704 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:39.195670 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:39.815154 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:39.815126 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6677bbd944-snbv2" Aug 05 16:28:40.898296 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:40.898263 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:44.400646 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:44.400611 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:44.501794 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:44.501759 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:45.255850 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.255814 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5"] Aug 05 16:28:45.264782 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.264763 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.268905 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.268887 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-g8dqh\"" Aug 05 16:28:45.269053 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.268967 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 05 16:28:45.273252 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.273231 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5"] Aug 05 16:28:45.291757 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.291727 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:45.434030 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.434000 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b2c4fa-00c7-4ee1-869e-756c25aad1b6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5\" (UID: \"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.434030 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.434035 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4fa-00c7-4ee1-869e-756c25aad1b6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5\" (UID: \"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.434570 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.434055 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5b2c4fa-00c7-4ee1-869e-756c25aad1b6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5\" (UID: \"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.434570 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.434102 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b2c4fa-00c7-4ee1-869e-756c25aad1b6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5\" (UID: \"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.434570 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.434137 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4fa-00c7-4ee1-869e-756c25aad1b6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5\" (UID: \"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.434570 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.434159 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4fa-00c7-4ee1-869e-756c25aad1b6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5\" (UID: \"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.434570 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.434187 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4fa-00c7-4ee1-869e-756c25aad1b6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5\" (UID: \"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.434570 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.434209 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhfrv\" (UniqueName: \"kubernetes.io/projected/d5b2c4fa-00c7-4ee1-869e-756c25aad1b6-kube-api-access-jhfrv\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5\" (UID: \"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.434570 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.434272 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4fa-00c7-4ee1-869e-756c25aad1b6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5\" (UID: \"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.535520 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.535483 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4fa-00c7-4ee1-869e-756c25aad1b6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5\" (UID: \"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.535705 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.535531 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b2c4fa-00c7-4ee1-869e-756c25aad1b6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5\" (UID: \"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.535705 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.535549 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4fa-00c7-4ee1-869e-756c25aad1b6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5\" (UID: \"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.535705 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.535567 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5b2c4fa-00c7-4ee1-869e-756c25aad1b6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5\" (UID: \"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.535705 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.535590 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b2c4fa-00c7-4ee1-869e-756c25aad1b6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5\" (UID: \"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.535705 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.535623 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4fa-00c7-4ee1-869e-756c25aad1b6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5\" (UID: \"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.535705 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.535652 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4fa-00c7-4ee1-869e-756c25aad1b6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5\" (UID: \"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.536008 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.535953 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4fa-00c7-4ee1-869e-756c25aad1b6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5\" (UID: \"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.536008 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.535962 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4fa-00c7-4ee1-869e-756c25aad1b6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5\" (UID: \"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.536008 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.535987 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhfrv\" (UniqueName: \"kubernetes.io/projected/d5b2c4fa-00c7-4ee1-869e-756c25aad1b6-kube-api-access-jhfrv\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5\" (UID: \"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.536240 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.536214 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4fa-00c7-4ee1-869e-756c25aad1b6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5\" (UID: \"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.536401 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.536339 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b2c4fa-00c7-4ee1-869e-756c25aad1b6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5\" (UID: \"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.536506 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.536465 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4fa-00c7-4ee1-869e-756c25aad1b6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5\" (UID: \"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.536781 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.536760 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4fa-00c7-4ee1-869e-756c25aad1b6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5\" (UID: \"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.538051 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.538021 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b2c4fa-00c7-4ee1-869e-756c25aad1b6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5\" (UID: \"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.538365 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.538346 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b2c4fa-00c7-4ee1-869e-756c25aad1b6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5\" (UID: \"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.539257 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.539230 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:45.547621 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.547596 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5b2c4fa-00c7-4ee1-869e-756c25aad1b6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5\" (UID: \"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.547899 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.547882 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhfrv\" (UniqueName: \"kubernetes.io/projected/d5b2c4fa-00c7-4ee1-869e-756c25aad1b6-kube-api-access-jhfrv\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5\" (UID: \"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.575763 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.575742 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:45.704333 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.704301 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5"] Aug 05 16:28:45.707780 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:28:45.707749 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b2c4fa_00c7_4ee1_869e_756c25aad1b6.slice/crio-80491781c4f9489d09772cf44cd6f80409992df45f717de81f32570ddb786b25 WatchSource:0}: Error finding container 80491781c4f9489d09772cf44cd6f80409992df45f717de81f32570ddb786b25: Status 404 returned error can't find the container with id 80491781c4f9489d09772cf44cd6f80409992df45f717de81f32570ddb786b25 Aug 05 16:28:45.825037 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.824943 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:28:45.869306 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:45.869276 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" event={"ID":"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6","Type":"ContainerStarted","Data":"80491781c4f9489d09772cf44cd6f80409992df45f717de81f32570ddb786b25"} Aug 05 16:28:48.940073 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:48.940038 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:28:48.940357 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:48.940112 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:28:48.940357 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:48.940142 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:28:49.887726 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:49.887693 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" event={"ID":"d5b2c4fa-00c7-4ee1-869e-756c25aad1b6","Type":"ContainerStarted","Data":"b962a8fc612ce128712c98097c375e5611f8d47008f43a1e9438ed9de08fcc44"} Aug 05 16:28:49.912101 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:49.912055 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" podStartSLOduration=1.682072186 podStartE2EDuration="4.912040818s" totalImagesPullingTime="3.229968632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:28:49.910639789 +0000 UTC m=+445.067485693" watchObservedRunningTime="2026-08-05 16:28:49.912040818 +0000 UTC m=+445.068886719" Aug 05 16:28:50.576108 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:50.576070 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:50.580700 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:50.580678 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:50.891803 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:50.891720 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:28:50.892658 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:28:50.892638 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5" Aug 05 16:29:27.160225 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:29:27.160141 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:27.254458 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:29:27.254421 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:29:44.586309 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:29:44.586275 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:03.613173 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:03.613122 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:05.690274 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:05.690236 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-sk7v8"] Aug 05 16:30:05.693272 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:05.693257 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-sk7v8" Aug 05 16:30:05.696553 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:05.696534 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 05 16:30:05.696653 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:05.696638 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-kztx5\"" Aug 05 16:30:05.698082 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:05.698065 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 05 16:30:05.710800 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:05.710777 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-sk7v8"] Aug 05 16:30:05.748941 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:05.748917 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xvtt\" (UniqueName: \"kubernetes.io/projected/00b0c5de-5743-42b3-8c32-13859778f9f9-kube-api-access-4xvtt\") pod \"limitador-operator-controller-manager-6c549b6d85-sk7v8\" (UID: \"00b0c5de-5743-42b3-8c32-13859778f9f9\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-sk7v8" Aug 05 16:30:05.850035 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:05.850006 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xvtt\" (UniqueName: \"kubernetes.io/projected/00b0c5de-5743-42b3-8c32-13859778f9f9-kube-api-access-4xvtt\") pod \"limitador-operator-controller-manager-6c549b6d85-sk7v8\" (UID: \"00b0c5de-5743-42b3-8c32-13859778f9f9\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-sk7v8" Aug 05 16:30:05.864054 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:05.864025 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xvtt\" (UniqueName: \"kubernetes.io/projected/00b0c5de-5743-42b3-8c32-13859778f9f9-kube-api-access-4xvtt\") pod \"limitador-operator-controller-manager-6c549b6d85-sk7v8\" (UID: \"00b0c5de-5743-42b3-8c32-13859778f9f9\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-sk7v8" Aug 05 16:30:06.003061 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:06.002976 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-sk7v8" Aug 05 16:30:06.141400 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:06.141274 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-sk7v8"] Aug 05 16:30:06.144035 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:30:06.143998 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b0c5de_5743_42b3_8c32_13859778f9f9.slice/crio-1f00bb8216ae1a95dfad7138a346e59f93ff889a86df6ad8b5be8f76649538e2 WatchSource:0}: Error finding container 1f00bb8216ae1a95dfad7138a346e59f93ff889a86df6ad8b5be8f76649538e2: Status 404 returned error can't find the container with id 1f00bb8216ae1a95dfad7138a346e59f93ff889a86df6ad8b5be8f76649538e2 Aug 05 16:30:06.173034 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:06.173005 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-sk7v8" event={"ID":"00b0c5de-5743-42b3-8c32-13859778f9f9","Type":"ContainerStarted","Data":"1f00bb8216ae1a95dfad7138a346e59f93ff889a86df6ad8b5be8f76649538e2"} Aug 05 16:30:06.421535 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:06.421507 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:08.039414 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:08.039360 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:08.258870 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:08.258837 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:08.431581 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:08.431498 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:08.553568 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:08.553530 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:09.189395 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:09.189297 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-sk7v8" event={"ID":"00b0c5de-5743-42b3-8c32-13859778f9f9","Type":"ContainerStarted","Data":"70cbcd883f28ec791b848b8cc9857a77bf239186698bd37b9293e920da121f0a"} Aug 05 16:30:09.189395 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:09.189361 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-sk7v8" Aug 05 16:30:09.213211 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:09.213163 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-sk7v8" podStartSLOduration=1.600404038 podStartE2EDuration="4.21314923s" totalImagesPullingTime="2.612745192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:30:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:30:09.211609362 +0000 UTC m=+524.368455304" watchObservedRunningTime="2026-08-05 16:30:09.21314923 +0000 UTC m=+524.369995127" Aug 05 16:30:09.727975 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:09.727945 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:10.883009 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:10.882968 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:10.987156 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:10.987080 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:11.086354 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:11.086326 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:11.619809 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:11.619753 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:11.902111 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:11.902002 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:12.079416 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:12.079324 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:12.177672 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:12.177584 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:15.196197 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:15.196165 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8"] Aug 05 16:30:15.199294 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:15.199279 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8" Aug 05 16:30:15.203827 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:15.203804 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vdhxn\"" Aug 05 16:30:15.221825 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:15.221796 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7zjh\" (UniqueName: \"kubernetes.io/projected/3df06b67-cac1-4ca9-96c1-e17103fbdb20-kube-api-access-b7zjh\") pod \"kuadrant-operator-controller-manager-58fc56757c-lkbx8\" (UID: \"3df06b67-cac1-4ca9-96c1-e17103fbdb20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8" Aug 05 16:30:15.221948 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:15.221841 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3df06b67-cac1-4ca9-96c1-e17103fbdb20-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-lkbx8\" (UID: \"3df06b67-cac1-4ca9-96c1-e17103fbdb20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8" Aug 05 16:30:15.229572 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:15.229547 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8"] Aug 05 16:30:15.322938 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:15.322908 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3df06b67-cac1-4ca9-96c1-e17103fbdb20-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-lkbx8\" (UID: \"3df06b67-cac1-4ca9-96c1-e17103fbdb20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8" Aug 05 16:30:15.323097 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:15.322966 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7zjh\" (UniqueName: \"kubernetes.io/projected/3df06b67-cac1-4ca9-96c1-e17103fbdb20-kube-api-access-b7zjh\") pod \"kuadrant-operator-controller-manager-58fc56757c-lkbx8\" (UID: \"3df06b67-cac1-4ca9-96c1-e17103fbdb20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8" Aug 05 16:30:15.323281 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:15.323263 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3df06b67-cac1-4ca9-96c1-e17103fbdb20-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-lkbx8\" (UID: \"3df06b67-cac1-4ca9-96c1-e17103fbdb20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8" Aug 05 16:30:15.336066 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:15.336044 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7zjh\" (UniqueName: \"kubernetes.io/projected/3df06b67-cac1-4ca9-96c1-e17103fbdb20-kube-api-access-b7zjh\") pod \"kuadrant-operator-controller-manager-58fc56757c-lkbx8\" (UID: \"3df06b67-cac1-4ca9-96c1-e17103fbdb20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8" Aug 05 16:30:15.509517 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:15.509486 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8" Aug 05 16:30:15.642289 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:15.642221 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8"] Aug 05 16:30:15.645034 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:30:15.645010 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df06b67_cac1_4ca9_96c1_e17103fbdb20.slice/crio-cae12e9a106b865630a94d4a042da39d23660f4cd6a985f1fd8f6945e6ed8679 WatchSource:0}: Error finding container cae12e9a106b865630a94d4a042da39d23660f4cd6a985f1fd8f6945e6ed8679: Status 404 returned error can't find the container with id cae12e9a106b865630a94d4a042da39d23660f4cd6a985f1fd8f6945e6ed8679 Aug 05 16:30:16.221154 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:16.221113 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8" event={"ID":"3df06b67-cac1-4ca9-96c1-e17103fbdb20","Type":"ContainerStarted","Data":"cae12e9a106b865630a94d4a042da39d23660f4cd6a985f1fd8f6945e6ed8679"} Aug 05 16:30:17.073748 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:17.073658 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:17.247067 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:17.247034 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:19.033299 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:19.033261 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:19.132553 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:19.132522 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:20.195941 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:20.195853 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-sk7v8" Aug 05 16:30:20.242110 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:20.242081 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8" event={"ID":"3df06b67-cac1-4ca9-96c1-e17103fbdb20","Type":"ContainerStarted","Data":"5868affba81a06bdcb1c26d81baa42b6d741c41e8de6b694a4de8ad84a942796"} Aug 05 16:30:20.242301 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:20.242216 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8" Aug 05 16:30:20.267364 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:20.267320 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8" podStartSLOduration=1.133084097 podStartE2EDuration="5.267305649s" totalImagesPullingTime="4.134221552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:30:20.265615584 +0000 UTC m=+535.422461498" watchObservedRunningTime="2026-08-05 16:30:20.267305649 +0000 UTC m=+535.424151553" Aug 05 16:30:22.424179 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:22.424145 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:22.623526 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:22.623492 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:25.061289 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:25.061254 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:25.161338 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:25.161304 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:29.051041 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:29.051006 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:29.658629 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:29.658595 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:29.733496 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:29.733466 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:29.826952 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:29.826918 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:29.927175 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:29.927095 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:30.028082 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:30.028044 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:31.247607 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:31.247577 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8" Aug 05 16:30:32.817873 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:32.817838 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-jwqpj"] Aug 05 16:30:32.823962 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:32.823945 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-jwqpj" Aug 05 16:30:32.843264 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:32.843235 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-jwqpj"] Aug 05 16:30:32.944269 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:32.944237 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-jwqpj"] Aug 05 16:30:32.944516 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:30:32.944497 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-pcbrk], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-jwqpj" podUID="3a619a98-c6e1-450b-8a31-e12633fdcca1" Aug 05 16:30:32.966802 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:32.966773 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8"] Aug 05 16:30:32.966963 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:32.966943 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8" podUID="3df06b67-cac1-4ca9-96c1-e17103fbdb20" containerName="manager" containerID="cri-o://5868affba81a06bdcb1c26d81baa42b6d741c41e8de6b694a4de8ad84a942796" gracePeriod=2 Aug 05 16:30:32.974840 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:32.974813 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a619a98-c6e1-450b-8a31-e12633fdcca1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-jwqpj\" (UID: \"3a619a98-c6e1-450b-8a31-e12633fdcca1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-jwqpj" Aug 05 16:30:32.974910 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:32.974866 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcbrk\" (UniqueName: \"kubernetes.io/projected/3a619a98-c6e1-450b-8a31-e12633fdcca1-kube-api-access-pcbrk\") pod \"kuadrant-operator-controller-manager-6bd956ccf-jwqpj\" (UID: \"3a619a98-c6e1-450b-8a31-e12633fdcca1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-jwqpj" Aug 05 16:30:32.985764 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:32.985742 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8"] Aug 05 16:30:32.998741 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:30:32.998703 2573 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3df06b67-cac1-4ca9-96c1-e17103fbdb20\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-58fc56757c-lkbx8\": pods \"kuadrant-operator-controller-manager-58fc56757c-lkbx8\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8" Aug 05 16:30:33.000169 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.000134 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rw5c8"] Aug 05 16:30:33.000591 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.000569 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3df06b67-cac1-4ca9-96c1-e17103fbdb20" containerName="manager" Aug 05 16:30:33.000591 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.000593 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3df06b67-cac1-4ca9-96c1-e17103fbdb20" containerName="manager" Aug 05 16:30:33.000757 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.000684 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3df06b67-cac1-4ca9-96c1-e17103fbdb20" containerName="manager" Aug 05 16:30:33.004471 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.004449 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rw5c8" Aug 05 16:30:33.005733 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.005707 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-jwqpj"] Aug 05 16:30:33.020611 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.020586 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-jwqpj"] Aug 05 16:30:33.022041 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.022020 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-b4892"] Aug 05 16:30:33.025794 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.025773 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rw5c8"] Aug 05 16:30:33.025903 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.025889 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-b4892" Aug 05 16:30:33.036455 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.036432 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-sk7v8"] Aug 05 16:30:33.036681 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.036654 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-sk7v8" podUID="00b0c5de-5743-42b3-8c32-13859778f9f9" containerName="manager" containerID="cri-o://70cbcd883f28ec791b848b8cc9857a77bf239186698bd37b9293e920da121f0a" gracePeriod=2 Aug 05 16:30:33.041079 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.040926 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-sk7v8"] Aug 05 16:30:33.061859 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.061786 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-b4892"] Aug 05 16:30:33.062423 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:30:33.062344 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-lkbx8\" is forbidden: User \"system:node:ip-10-0-132-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-37.ec2.internal' and this object" podUID="3df06b67-cac1-4ca9-96c1-e17103fbdb20" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8" Aug 05 16:30:33.065325 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:30:33.065301 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-lkbx8\" is forbidden: User \"system:node:ip-10-0-132-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-37.ec2.internal' and this object" podUID="3df06b67-cac1-4ca9-96c1-e17103fbdb20" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8" Aug 05 16:30:33.075332 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.075267 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a619a98-c6e1-450b-8a31-e12633fdcca1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-jwqpj\" (UID: \"3a619a98-c6e1-450b-8a31-e12633fdcca1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-jwqpj" Aug 05 16:30:33.075332 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.075310 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/734db8c4-36a6-47ee-b61e-a5d77e7354b3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-rw5c8\" (UID: \"734db8c4-36a6-47ee-b61e-a5d77e7354b3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rw5c8" Aug 05 16:30:33.075520 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.075353 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcbrk\" (UniqueName: \"kubernetes.io/projected/3a619a98-c6e1-450b-8a31-e12633fdcca1-kube-api-access-pcbrk\") pod \"kuadrant-operator-controller-manager-6bd956ccf-jwqpj\" (UID: \"3a619a98-c6e1-450b-8a31-e12633fdcca1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-jwqpj" Aug 05 16:30:33.075520 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.075446 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpcbt\" (UniqueName: \"kubernetes.io/projected/734db8c4-36a6-47ee-b61e-a5d77e7354b3-kube-api-access-gpcbt\") pod \"kuadrant-operator-controller-manager-68cf589589-rw5c8\" (UID: \"734db8c4-36a6-47ee-b61e-a5d77e7354b3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rw5c8" Aug 05 16:30:33.075875 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.075852 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a619a98-c6e1-450b-8a31-e12633fdcca1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-jwqpj\" (UID: \"3a619a98-c6e1-450b-8a31-e12633fdcca1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-jwqpj" Aug 05 16:30:33.082594 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.082574 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-p6m27"] Aug 05 16:30:33.082846 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.082835 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00b0c5de-5743-42b3-8c32-13859778f9f9" containerName="manager" Aug 05 16:30:33.082882 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.082848 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="00b0c5de-5743-42b3-8c32-13859778f9f9" containerName="manager" Aug 05 16:30:33.082917 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.082900 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="00b0c5de-5743-42b3-8c32-13859778f9f9" containerName="manager" Aug 05 16:30:33.084775 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:30:33.084755 2573 projected.go:196] Error preparing data for projected volume kube-api-access-pcbrk for pod kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-jwqpj: failed to fetch token: pod "kuadrant-operator-controller-manager-6bd956ccf-jwqpj" not found Aug 05 16:30:33.084864 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:30:33.084852 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a619a98-c6e1-450b-8a31-e12633fdcca1-kube-api-access-pcbrk podName:3a619a98-c6e1-450b-8a31-e12633fdcca1 nodeName:}" failed. No retries permitted until 2026-08-05 16:30:33.584832244 +0000 UTC m=+548.741678127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pcbrk" (UniqueName: "kubernetes.io/projected/3a619a98-c6e1-450b-8a31-e12633fdcca1-kube-api-access-pcbrk") pod "kuadrant-operator-controller-manager-6bd956ccf-jwqpj" (UID: "3a619a98-c6e1-450b-8a31-e12633fdcca1") : failed to fetch token: pod "kuadrant-operator-controller-manager-6bd956ccf-jwqpj" not found Aug 05 16:30:33.085826 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.085811 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-p6m27" Aug 05 16:30:33.092248 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:30:33.092222 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-sk7v8\" is forbidden: User \"system:node:ip-10-0-132-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-37.ec2.internal' and this object" podUID="00b0c5de-5743-42b3-8c32-13859778f9f9" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-sk7v8" Aug 05 16:30:33.095874 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:30:33.095850 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-sk7v8\" is forbidden: User \"system:node:ip-10-0-132-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-37.ec2.internal' and this object" podUID="00b0c5de-5743-42b3-8c32-13859778f9f9" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-sk7v8" Aug 05 16:30:33.098892 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:30:33.098852 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-lkbx8\" is forbidden: User \"system:node:ip-10-0-132-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-37.ec2.internal' and this object" podUID="3df06b67-cac1-4ca9-96c1-e17103fbdb20" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8" Aug 05 16:30:33.100370 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.100345 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-p6m27"] Aug 05 16:30:33.176450 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.176417 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb5zs\" (UniqueName: \"kubernetes.io/projected/a28486bd-37b1-4458-8623-a407b50d5e86-kube-api-access-hb5zs\") pod \"kuadrant-operator-controller-manager-58fc56757c-b4892\" (UID: \"a28486bd-37b1-4458-8623-a407b50d5e86\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-b4892" Aug 05 16:30:33.176616 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.176482 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/734db8c4-36a6-47ee-b61e-a5d77e7354b3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-rw5c8\" (UID: \"734db8c4-36a6-47ee-b61e-a5d77e7354b3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rw5c8" Aug 05 16:30:33.176616 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.176531 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a28486bd-37b1-4458-8623-a407b50d5e86-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-b4892\" (UID: \"a28486bd-37b1-4458-8623-a407b50d5e86\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-b4892" Aug 05 16:30:33.176758 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.176699 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpcbt\" (UniqueName: \"kubernetes.io/projected/734db8c4-36a6-47ee-b61e-a5d77e7354b3-kube-api-access-gpcbt\") pod \"kuadrant-operator-controller-manager-68cf589589-rw5c8\" (UID: \"734db8c4-36a6-47ee-b61e-a5d77e7354b3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rw5c8" Aug 05 16:30:33.180412 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.177055 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df78h\" (UniqueName: \"kubernetes.io/projected/07f3e42e-2c78-4ac1-8b67-dd747f0326d3-kube-api-access-df78h\") pod \"limitador-operator-controller-manager-6c549b6d85-p6m27\" (UID: \"07f3e42e-2c78-4ac1-8b67-dd747f0326d3\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-p6m27" Aug 05 16:30:33.180412 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.177504 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/734db8c4-36a6-47ee-b61e-a5d77e7354b3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-rw5c8\" (UID: \"734db8c4-36a6-47ee-b61e-a5d77e7354b3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rw5c8" Aug 05 16:30:33.188013 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.187979 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpcbt\" (UniqueName: \"kubernetes.io/projected/734db8c4-36a6-47ee-b61e-a5d77e7354b3-kube-api-access-gpcbt\") pod \"kuadrant-operator-controller-manager-68cf589589-rw5c8\" (UID: \"734db8c4-36a6-47ee-b61e-a5d77e7354b3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rw5c8" Aug 05 16:30:33.270528 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.270505 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8" Aug 05 16:30:33.272890 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.272872 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-sk7v8" Aug 05 16:30:33.273581 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:30:33.273559 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-sk7v8\" is forbidden: User \"system:node:ip-10-0-132-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-37.ec2.internal' and this object" podUID="00b0c5de-5743-42b3-8c32-13859778f9f9" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-sk7v8" Aug 05 16:30:33.276359 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:30:33.276338 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-lkbx8\" is forbidden: User \"system:node:ip-10-0-132-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-37.ec2.internal' and this object" podUID="3df06b67-cac1-4ca9-96c1-e17103fbdb20" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8" Aug 05 16:30:33.277704 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.277689 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df78h\" (UniqueName: \"kubernetes.io/projected/07f3e42e-2c78-4ac1-8b67-dd747f0326d3-kube-api-access-df78h\") pod \"limitador-operator-controller-manager-6c549b6d85-p6m27\" (UID: \"07f3e42e-2c78-4ac1-8b67-dd747f0326d3\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-p6m27" Aug 05 16:30:33.277759 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.277722 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb5zs\" (UniqueName: \"kubernetes.io/projected/a28486bd-37b1-4458-8623-a407b50d5e86-kube-api-access-hb5zs\") pod \"kuadrant-operator-controller-manager-58fc56757c-b4892\" (UID: \"a28486bd-37b1-4458-8623-a407b50d5e86\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-b4892" Aug 05 16:30:33.277816 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.277765 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a28486bd-37b1-4458-8623-a407b50d5e86-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-b4892\" (UID: \"a28486bd-37b1-4458-8623-a407b50d5e86\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-b4892" Aug 05 16:30:33.278083 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.278067 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a28486bd-37b1-4458-8623-a407b50d5e86-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-b4892\" (UID: \"a28486bd-37b1-4458-8623-a407b50d5e86\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-b4892" Aug 05 16:30:33.278671 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:30:33.278653 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-sk7v8\" is forbidden: User \"system:node:ip-10-0-132-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-37.ec2.internal' and this object" podUID="00b0c5de-5743-42b3-8c32-13859778f9f9" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-sk7v8" Aug 05 16:30:33.281036 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:30:33.281017 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-lkbx8\" is forbidden: User \"system:node:ip-10-0-132-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-37.ec2.internal' and this object" podUID="3df06b67-cac1-4ca9-96c1-e17103fbdb20" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8" Aug 05 16:30:33.287415 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.287396 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df78h\" (UniqueName: \"kubernetes.io/projected/07f3e42e-2c78-4ac1-8b67-dd747f0326d3-kube-api-access-df78h\") pod \"limitador-operator-controller-manager-6c549b6d85-p6m27\" (UID: \"07f3e42e-2c78-4ac1-8b67-dd747f0326d3\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-p6m27" Aug 05 16:30:33.287502 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.287407 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb5zs\" (UniqueName: \"kubernetes.io/projected/a28486bd-37b1-4458-8623-a407b50d5e86-kube-api-access-hb5zs\") pod \"kuadrant-operator-controller-manager-58fc56757c-b4892\" (UID: \"a28486bd-37b1-4458-8623-a407b50d5e86\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-b4892" Aug 05 16:30:33.295345 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.295324 2573 generic.go:356] "Generic (PLEG): container finished" podID="00b0c5de-5743-42b3-8c32-13859778f9f9" containerID="70cbcd883f28ec791b848b8cc9857a77bf239186698bd37b9293e920da121f0a" exitCode=0 Aug 05 16:30:33.295530 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.295370 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-sk7v8" Aug 05 16:30:33.295530 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.295434 2573 scope.go:117] "RemoveContainer" containerID="70cbcd883f28ec791b848b8cc9857a77bf239186698bd37b9293e920da121f0a" Aug 05 16:30:33.296806 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.296785 2573 generic.go:356] "Generic (PLEG): container finished" podID="3df06b67-cac1-4ca9-96c1-e17103fbdb20" containerID="5868affba81a06bdcb1c26d81baa42b6d741c41e8de6b694a4de8ad84a942796" exitCode=0 Aug 05 16:30:33.296884 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.296822 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8" Aug 05 16:30:33.296955 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.296944 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-jwqpj" Aug 05 16:30:33.298486 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:30:33.298468 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-lkbx8\" is forbidden: User \"system:node:ip-10-0-132-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-37.ec2.internal' and this object" podUID="3df06b67-cac1-4ca9-96c1-e17103fbdb20" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8" Aug 05 16:30:33.301180 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:30:33.301158 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-sk7v8\" is forbidden: User \"system:node:ip-10-0-132-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-37.ec2.internal' and this object" podUID="00b0c5de-5743-42b3-8c32-13859778f9f9" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-sk7v8" Aug 05 16:30:33.303535 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.303516 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-jwqpj" Aug 05 16:30:33.303771 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:30:33.303750 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-lkbx8\" is forbidden: User \"system:node:ip-10-0-132-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-37.ec2.internal' and this object" podUID="3df06b67-cac1-4ca9-96c1-e17103fbdb20" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-lkbx8" Aug 05 16:30:33.305102 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.305079 2573 scope.go:117] "RemoveContainer" containerID="70cbcd883f28ec791b848b8cc9857a77bf239186698bd37b9293e920da121f0a" Aug 05 16:30:33.305353 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:30:33.305333 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70cbcd883f28ec791b848b8cc9857a77bf239186698bd37b9293e920da121f0a\": container with ID starting with 70cbcd883f28ec791b848b8cc9857a77bf239186698bd37b9293e920da121f0a not found: ID does not exist" containerID="70cbcd883f28ec791b848b8cc9857a77bf239186698bd37b9293e920da121f0a" Aug 05 16:30:33.305458 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.305366 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70cbcd883f28ec791b848b8cc9857a77bf239186698bd37b9293e920da121f0a"} err="failed to get container status \"70cbcd883f28ec791b848b8cc9857a77bf239186698bd37b9293e920da121f0a\": rpc error: code = NotFound desc = could not find container \"70cbcd883f28ec791b848b8cc9857a77bf239186698bd37b9293e920da121f0a\": container with ID starting with 70cbcd883f28ec791b848b8cc9857a77bf239186698bd37b9293e920da121f0a not found: ID does not exist" Aug 05 16:30:33.305458 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.305416 2573 scope.go:117] "RemoveContainer" containerID="5868affba81a06bdcb1c26d81baa42b6d741c41e8de6b694a4de8ad84a942796" Aug 05 16:30:33.306563 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:30:33.306540 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-sk7v8\" is forbidden: User \"system:node:ip-10-0-132-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-37.ec2.internal' and this object" podUID="00b0c5de-5743-42b3-8c32-13859778f9f9" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-sk7v8" Aug 05 16:30:33.309049 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:30:33.309029 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-jwqpj\" is forbidden: User \"system:node:ip-10-0-132-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-37.ec2.internal' and this object" podUID="3a619a98-c6e1-450b-8a31-e12633fdcca1" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-jwqpj" Aug 05 16:30:33.312425 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.312406 2573 scope.go:117] "RemoveContainer" containerID="5868affba81a06bdcb1c26d81baa42b6d741c41e8de6b694a4de8ad84a942796" Aug 05 16:30:33.312639 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:30:33.312622 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5868affba81a06bdcb1c26d81baa42b6d741c41e8de6b694a4de8ad84a942796\": container with ID starting with 5868affba81a06bdcb1c26d81baa42b6d741c41e8de6b694a4de8ad84a942796 not found: ID does not exist" containerID="5868affba81a06bdcb1c26d81baa42b6d741c41e8de6b694a4de8ad84a942796" Aug 05 16:30:33.312684 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.312644 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5868affba81a06bdcb1c26d81baa42b6d741c41e8de6b694a4de8ad84a942796"} err="failed to get container status \"5868affba81a06bdcb1c26d81baa42b6d741c41e8de6b694a4de8ad84a942796\": rpc error: code = NotFound desc = could not find container \"5868affba81a06bdcb1c26d81baa42b6d741c41e8de6b694a4de8ad84a942796\": container with ID starting with 5868affba81a06bdcb1c26d81baa42b6d741c41e8de6b694a4de8ad84a942796 not found: ID does not exist" Aug 05 16:30:33.344421 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.344356 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rw5c8" Aug 05 16:30:33.378583 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.378550 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xvtt\" (UniqueName: \"kubernetes.io/projected/00b0c5de-5743-42b3-8c32-13859778f9f9-kube-api-access-4xvtt\") pod \"00b0c5de-5743-42b3-8c32-13859778f9f9\" (UID: \"00b0c5de-5743-42b3-8c32-13859778f9f9\") " Aug 05 16:30:33.378583 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.378590 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a619a98-c6e1-450b-8a31-e12633fdcca1-extensions-socket-volume\") pod \"3a619a98-c6e1-450b-8a31-e12633fdcca1\" (UID: \"3a619a98-c6e1-450b-8a31-e12633fdcca1\") " Aug 05 16:30:33.378748 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.378612 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7zjh\" (UniqueName: \"kubernetes.io/projected/3df06b67-cac1-4ca9-96c1-e17103fbdb20-kube-api-access-b7zjh\") pod \"3df06b67-cac1-4ca9-96c1-e17103fbdb20\" (UID: \"3df06b67-cac1-4ca9-96c1-e17103fbdb20\") " Aug 05 16:30:33.378748 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.378650 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3df06b67-cac1-4ca9-96c1-e17103fbdb20-extensions-socket-volume\") pod \"3df06b67-cac1-4ca9-96c1-e17103fbdb20\" (UID: \"3df06b67-cac1-4ca9-96c1-e17103fbdb20\") " Aug 05 16:30:33.378885 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.378859 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a619a98-c6e1-450b-8a31-e12633fdcca1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3a619a98-c6e1-450b-8a31-e12633fdcca1" (UID: "3a619a98-c6e1-450b-8a31-e12633fdcca1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:30:33.378929 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.378883 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcbrk\" (UniqueName: \"kubernetes.io/projected/3a619a98-c6e1-450b-8a31-e12633fdcca1-kube-api-access-pcbrk\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:30:33.379130 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.379106 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df06b67-cac1-4ca9-96c1-e17103fbdb20-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3df06b67-cac1-4ca9-96c1-e17103fbdb20" (UID: "3df06b67-cac1-4ca9-96c1-e17103fbdb20"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:30:33.380781 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.380761 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b0c5de-5743-42b3-8c32-13859778f9f9-kube-api-access-4xvtt" (OuterVolumeSpecName: "kube-api-access-4xvtt") pod "00b0c5de-5743-42b3-8c32-13859778f9f9" (UID: "00b0c5de-5743-42b3-8c32-13859778f9f9"). InnerVolumeSpecName "kube-api-access-4xvtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:30:33.380882 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.380811 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df06b67-cac1-4ca9-96c1-e17103fbdb20-kube-api-access-b7zjh" (OuterVolumeSpecName: "kube-api-access-b7zjh") pod "3df06b67-cac1-4ca9-96c1-e17103fbdb20" (UID: "3df06b67-cac1-4ca9-96c1-e17103fbdb20"). InnerVolumeSpecName "kube-api-access-b7zjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:30:33.406053 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.406022 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-b4892" Aug 05 16:30:33.411224 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.411194 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-p6m27" Aug 05 16:30:33.480406 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.480291 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xvtt\" (UniqueName: \"kubernetes.io/projected/00b0c5de-5743-42b3-8c32-13859778f9f9-kube-api-access-4xvtt\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:30:33.480406 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.480324 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a619a98-c6e1-450b-8a31-e12633fdcca1-extensions-socket-volume\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:30:33.480406 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.480339 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7zjh\" (UniqueName: \"kubernetes.io/projected/3df06b67-cac1-4ca9-96c1-e17103fbdb20-kube-api-access-b7zjh\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:30:33.480406 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.480354 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3df06b67-cac1-4ca9-96c1-e17103fbdb20-extensions-socket-volume\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:30:33.494172 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.494141 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rw5c8"] Aug 05 16:30:33.496267 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:30:33.496245 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734db8c4_36a6_47ee_b61e_a5d77e7354b3.slice/crio-e39d1a7f3911b51f4c263a2e0b81b7c8ff7a13cf47f5721cee022b3ccd595201 WatchSource:0}: Error finding container e39d1a7f3911b51f4c263a2e0b81b7c8ff7a13cf47f5721cee022b3ccd595201: Status 404 returned error can't find the container with id e39d1a7f3911b51f4c263a2e0b81b7c8ff7a13cf47f5721cee022b3ccd595201 Aug 05 16:30:33.560867 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.560845 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-b4892"] Aug 05 16:30:33.563500 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:30:33.563476 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda28486bd_37b1_4458_8623_a407b50d5e86.slice/crio-0301c6ca314c3510935d980673f345661a79f8ab18307698c452cf1d56ff70fc WatchSource:0}: Error finding container 0301c6ca314c3510935d980673f345661a79f8ab18307698c452cf1d56ff70fc: Status 404 returned error can't find the container with id 0301c6ca314c3510935d980673f345661a79f8ab18307698c452cf1d56ff70fc Aug 05 16:30:33.576790 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:33.576728 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-p6m27"] Aug 05 16:30:33.579932 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:30:33.579902 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f3e42e_2c78_4ac1_8b67_dd747f0326d3.slice/crio-aedd88d4b4f0744cb9e529ba3d6e84322367616d7778675ceeed90590df8f6c5 WatchSource:0}: Error finding container aedd88d4b4f0744cb9e529ba3d6e84322367616d7778675ceeed90590df8f6c5: Status 404 returned error can't find the container with id aedd88d4b4f0744cb9e529ba3d6e84322367616d7778675ceeed90590df8f6c5 Aug 05 16:30:34.303069 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:34.303038 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-p6m27" event={"ID":"07f3e42e-2c78-4ac1-8b67-dd747f0326d3","Type":"ContainerStarted","Data":"a489c6de85d2eb57c25759750010a414a1079d453899a821affff6db69f8f24d"} Aug 05 16:30:34.303588 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:34.303193 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-p6m27" Aug 05 16:30:34.303588 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:34.303234 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-p6m27" event={"ID":"07f3e42e-2c78-4ac1-8b67-dd747f0326d3","Type":"ContainerStarted","Data":"aedd88d4b4f0744cb9e529ba3d6e84322367616d7778675ceeed90590df8f6c5"} Aug 05 16:30:34.305553 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:34.305530 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-b4892" event={"ID":"a28486bd-37b1-4458-8623-a407b50d5e86","Type":"ContainerStarted","Data":"16bd034d3f6f0285b4fe1f218c57e68a0e316cf9cd8af2da5d4bcd64d68e7567"} Aug 05 16:30:34.305690 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:34.305573 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-b4892" event={"ID":"a28486bd-37b1-4458-8623-a407b50d5e86","Type":"ContainerStarted","Data":"0301c6ca314c3510935d980673f345661a79f8ab18307698c452cf1d56ff70fc"} Aug 05 16:30:34.305690 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:34.305600 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-b4892" Aug 05 16:30:34.307021 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:34.306998 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rw5c8" event={"ID":"734db8c4-36a6-47ee-b61e-a5d77e7354b3","Type":"ContainerStarted","Data":"0dd26bfa89c4a5d7f803e17b706fe3d095ccddfc690598ba0e5f640e1a206e6e"} Aug 05 16:30:34.307123 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:34.307035 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rw5c8" event={"ID":"734db8c4-36a6-47ee-b61e-a5d77e7354b3","Type":"ContainerStarted","Data":"e39d1a7f3911b51f4c263a2e0b81b7c8ff7a13cf47f5721cee022b3ccd595201"} Aug 05 16:30:34.307123 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:34.307073 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rw5c8" Aug 05 16:30:34.307877 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:34.307859 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-jwqpj" Aug 05 16:30:34.328145 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:34.327605 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-p6m27" podStartSLOduration=1.3275866619999999 podStartE2EDuration="1.327586662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:30:34.325668271 +0000 UTC m=+549.482514171" watchObservedRunningTime="2026-08-05 16:30:34.327586662 +0000 UTC m=+549.484432564" Aug 05 16:30:34.352991 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:34.352944 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-b4892" podStartSLOduration=2.352929782 podStartE2EDuration="2.352929782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:30:34.352494569 +0000 UTC m=+549.509340469" watchObservedRunningTime="2026-08-05 16:30:34.352929782 +0000 UTC m=+549.509775682" Aug 05 16:30:34.355035 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:30:34.355011 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-jwqpj\" is forbidden: User \"system:node:ip-10-0-132-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-37.ec2.internal' and this object" podUID="3a619a98-c6e1-450b-8a31-e12633fdcca1" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-jwqpj" Aug 05 16:30:34.387106 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:34.387063 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rw5c8" podStartSLOduration=2.387048975 podStartE2EDuration="2.387048975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:30:34.385449833 +0000 UTC m=+549.542295734" watchObservedRunningTime="2026-08-05 16:30:34.387048975 +0000 UTC m=+549.543894876" Aug 05 16:30:35.302501 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:30:35.302466 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-jwqpj\" is forbidden: User \"system:node:ip-10-0-132-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-37.ec2.internal' and this object" podUID="3a619a98-c6e1-450b-8a31-e12633fdcca1" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-jwqpj" Aug 05 16:30:35.303205 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:35.303185 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00b0c5de-5743-42b3-8c32-13859778f9f9" path="/var/lib/kubelet/pods/00b0c5de-5743-42b3-8c32-13859778f9f9/volumes" Aug 05 16:30:35.303512 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:35.303500 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a619a98-c6e1-450b-8a31-e12633fdcca1" path="/var/lib/kubelet/pods/3a619a98-c6e1-450b-8a31-e12633fdcca1/volumes" Aug 05 16:30:35.303713 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:35.303703 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3df06b67-cac1-4ca9-96c1-e17103fbdb20" path="/var/lib/kubelet/pods/3df06b67-cac1-4ca9-96c1-e17103fbdb20/volumes" Aug 05 16:30:36.710752 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:36.710719 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:36.823033 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:36.822971 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:39.116830 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:39.116798 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:39.278687 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:39.278650 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:44.174950 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:44.174910 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:44.274935 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:44.274902 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:45.313958 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:45.313933 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-b4892" Aug 05 16:30:45.314401 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:45.313998 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-p6m27" Aug 05 16:30:45.314401 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:45.314147 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rw5c8" Aug 05 16:30:45.433431 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:45.433402 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-b4892"] Aug 05 16:30:45.433676 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:45.433632 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-b4892" podUID="a28486bd-37b1-4458-8623-a407b50d5e86" containerName="manager" containerID="cri-o://16bd034d3f6f0285b4fe1f218c57e68a0e316cf9cd8af2da5d4bcd64d68e7567" gracePeriod=10 Aug 05 16:30:45.673984 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:45.673963 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-b4892" Aug 05 16:30:45.783751 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:45.783721 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hb5zs\" (UniqueName: \"kubernetes.io/projected/a28486bd-37b1-4458-8623-a407b50d5e86-kube-api-access-hb5zs\") pod \"a28486bd-37b1-4458-8623-a407b50d5e86\" (UID: \"a28486bd-37b1-4458-8623-a407b50d5e86\") " Aug 05 16:30:45.783906 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:45.783806 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a28486bd-37b1-4458-8623-a407b50d5e86-extensions-socket-volume\") pod \"a28486bd-37b1-4458-8623-a407b50d5e86\" (UID: \"a28486bd-37b1-4458-8623-a407b50d5e86\") " Aug 05 16:30:45.784180 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:45.784155 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28486bd-37b1-4458-8623-a407b50d5e86-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a28486bd-37b1-4458-8623-a407b50d5e86" (UID: "a28486bd-37b1-4458-8623-a407b50d5e86"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:30:45.785742 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:45.785720 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a28486bd-37b1-4458-8623-a407b50d5e86-kube-api-access-hb5zs" (OuterVolumeSpecName: "kube-api-access-hb5zs") pod "a28486bd-37b1-4458-8623-a407b50d5e86" (UID: "a28486bd-37b1-4458-8623-a407b50d5e86"). InnerVolumeSpecName "kube-api-access-hb5zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:30:45.885029 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:45.884944 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb5zs\" (UniqueName: \"kubernetes.io/projected/a28486bd-37b1-4458-8623-a407b50d5e86-kube-api-access-hb5zs\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:30:45.885029 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:45.884972 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a28486bd-37b1-4458-8623-a407b50d5e86-extensions-socket-volume\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:30:46.360172 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:46.360136 2573 generic.go:356] "Generic (PLEG): container finished" podID="a28486bd-37b1-4458-8623-a407b50d5e86" containerID="16bd034d3f6f0285b4fe1f218c57e68a0e316cf9cd8af2da5d4bcd64d68e7567" exitCode=0 Aug 05 16:30:46.360631 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:46.360183 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-b4892" Aug 05 16:30:46.360631 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:46.360227 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-b4892" event={"ID":"a28486bd-37b1-4458-8623-a407b50d5e86","Type":"ContainerDied","Data":"16bd034d3f6f0285b4fe1f218c57e68a0e316cf9cd8af2da5d4bcd64d68e7567"} Aug 05 16:30:46.360631 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:46.360281 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-b4892" event={"ID":"a28486bd-37b1-4458-8623-a407b50d5e86","Type":"ContainerDied","Data":"0301c6ca314c3510935d980673f345661a79f8ab18307698c452cf1d56ff70fc"} Aug 05 16:30:46.360631 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:46.360302 2573 scope.go:117] "RemoveContainer" containerID="16bd034d3f6f0285b4fe1f218c57e68a0e316cf9cd8af2da5d4bcd64d68e7567" Aug 05 16:30:46.369054 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:46.369039 2573 scope.go:117] "RemoveContainer" containerID="16bd034d3f6f0285b4fe1f218c57e68a0e316cf9cd8af2da5d4bcd64d68e7567" Aug 05 16:30:46.369310 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:30:46.369294 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16bd034d3f6f0285b4fe1f218c57e68a0e316cf9cd8af2da5d4bcd64d68e7567\": container with ID starting with 16bd034d3f6f0285b4fe1f218c57e68a0e316cf9cd8af2da5d4bcd64d68e7567 not found: ID does not exist" containerID="16bd034d3f6f0285b4fe1f218c57e68a0e316cf9cd8af2da5d4bcd64d68e7567" Aug 05 16:30:46.369478 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:46.369316 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16bd034d3f6f0285b4fe1f218c57e68a0e316cf9cd8af2da5d4bcd64d68e7567"} err="failed to get container status \"16bd034d3f6f0285b4fe1f218c57e68a0e316cf9cd8af2da5d4bcd64d68e7567\": rpc error: code = NotFound desc = could not find container \"16bd034d3f6f0285b4fe1f218c57e68a0e316cf9cd8af2da5d4bcd64d68e7567\": container with ID starting with 16bd034d3f6f0285b4fe1f218c57e68a0e316cf9cd8af2da5d4bcd64d68e7567 not found: ID does not exist" Aug 05 16:30:46.386926 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:46.386904 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-b4892"] Aug 05 16:30:46.400931 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:46.400909 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-b4892"] Aug 05 16:30:47.303532 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:47.303496 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a28486bd-37b1-4458-8623-a407b50d5e86" path="/var/lib/kubelet/pods/a28486bd-37b1-4458-8623-a407b50d5e86/volumes" Aug 05 16:30:49.022586 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:49.022556 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:49.165871 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:49.165804 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:30:49.269160 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:30:49.269122 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:01.762130 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.762101 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj"] Aug 05 16:31:01.762581 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.762414 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a28486bd-37b1-4458-8623-a407b50d5e86" containerName="manager" Aug 05 16:31:01.762581 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.762426 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a28486bd-37b1-4458-8623-a407b50d5e86" containerName="manager" Aug 05 16:31:01.762581 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.762487 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a28486bd-37b1-4458-8623-a407b50d5e86" containerName="manager" Aug 05 16:31:01.764348 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.764333 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:01.767144 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.767126 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pwzsx\"" Aug 05 16:31:01.782966 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.782940 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj"] Aug 05 16:31:01.799566 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.799535 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:01.810284 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.810253 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c8f3579-b836-485b-923c-21f2ba37336a-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-x7llj\" (UID: \"4c8f3579-b836-485b-923c-21f2ba37336a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:01.810434 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.810285 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c8f3579-b836-485b-923c-21f2ba37336a-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-x7llj\" (UID: \"4c8f3579-b836-485b-923c-21f2ba37336a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:01.810434 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.810307 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c8f3579-b836-485b-923c-21f2ba37336a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-x7llj\" (UID: \"4c8f3579-b836-485b-923c-21f2ba37336a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:01.810434 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.810327 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c8f3579-b836-485b-923c-21f2ba37336a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-x7llj\" (UID: \"4c8f3579-b836-485b-923c-21f2ba37336a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:01.810612 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.810424 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8f3579-b836-485b-923c-21f2ba37336a-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-x7llj\" (UID: \"4c8f3579-b836-485b-923c-21f2ba37336a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:01.810612 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.810486 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c8f3579-b836-485b-923c-21f2ba37336a-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-x7llj\" (UID: \"4c8f3579-b836-485b-923c-21f2ba37336a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:01.810612 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.810528 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8f3579-b836-485b-923c-21f2ba37336a-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-x7llj\" (UID: \"4c8f3579-b836-485b-923c-21f2ba37336a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:01.810612 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.810558 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c8f3579-b836-485b-923c-21f2ba37336a-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-x7llj\" (UID: \"4c8f3579-b836-485b-923c-21f2ba37336a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:01.810612 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.810602 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsvh7\" (UniqueName: \"kubernetes.io/projected/4c8f3579-b836-485b-923c-21f2ba37336a-kube-api-access-vsvh7\") pod \"maas-default-gateway-openshift-default-78bb78ff78-x7llj\" (UID: \"4c8f3579-b836-485b-923c-21f2ba37336a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:01.911529 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.911491 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c8f3579-b836-485b-923c-21f2ba37336a-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-x7llj\" (UID: \"4c8f3579-b836-485b-923c-21f2ba37336a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:01.911696 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.911537 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8f3579-b836-485b-923c-21f2ba37336a-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-x7llj\" (UID: \"4c8f3579-b836-485b-923c-21f2ba37336a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:01.911696 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.911566 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c8f3579-b836-485b-923c-21f2ba37336a-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-x7llj\" (UID: \"4c8f3579-b836-485b-923c-21f2ba37336a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:01.911696 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.911604 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsvh7\" (UniqueName: \"kubernetes.io/projected/4c8f3579-b836-485b-923c-21f2ba37336a-kube-api-access-vsvh7\") pod \"maas-default-gateway-openshift-default-78bb78ff78-x7llj\" (UID: \"4c8f3579-b836-485b-923c-21f2ba37336a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:01.911696 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.911642 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c8f3579-b836-485b-923c-21f2ba37336a-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-x7llj\" (UID: \"4c8f3579-b836-485b-923c-21f2ba37336a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:01.911696 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.911668 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c8f3579-b836-485b-923c-21f2ba37336a-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-x7llj\" (UID: \"4c8f3579-b836-485b-923c-21f2ba37336a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:01.911696 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.911696 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c8f3579-b836-485b-923c-21f2ba37336a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-x7llj\" (UID: \"4c8f3579-b836-485b-923c-21f2ba37336a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:01.912029 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.911729 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c8f3579-b836-485b-923c-21f2ba37336a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-x7llj\" (UID: \"4c8f3579-b836-485b-923c-21f2ba37336a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:01.912029 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.911760 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8f3579-b836-485b-923c-21f2ba37336a-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-x7llj\" (UID: \"4c8f3579-b836-485b-923c-21f2ba37336a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:01.912134 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.912074 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c8f3579-b836-485b-923c-21f2ba37336a-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-x7llj\" (UID: \"4c8f3579-b836-485b-923c-21f2ba37336a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:01.912239 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.912215 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8f3579-b836-485b-923c-21f2ba37336a-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-x7llj\" (UID: \"4c8f3579-b836-485b-923c-21f2ba37336a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:01.912405 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.912361 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c8f3579-b836-485b-923c-21f2ba37336a-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-x7llj\" (UID: \"4c8f3579-b836-485b-923c-21f2ba37336a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:01.912405 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.912394 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8f3579-b836-485b-923c-21f2ba37336a-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-x7llj\" (UID: \"4c8f3579-b836-485b-923c-21f2ba37336a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:01.912538 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.912496 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c8f3579-b836-485b-923c-21f2ba37336a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-x7llj\" (UID: \"4c8f3579-b836-485b-923c-21f2ba37336a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:01.914082 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.914056 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c8f3579-b836-485b-923c-21f2ba37336a-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-x7llj\" (UID: \"4c8f3579-b836-485b-923c-21f2ba37336a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:01.914295 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.914276 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c8f3579-b836-485b-923c-21f2ba37336a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-x7llj\" (UID: \"4c8f3579-b836-485b-923c-21f2ba37336a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:01.920916 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.920884 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c8f3579-b836-485b-923c-21f2ba37336a-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-x7llj\" (UID: \"4c8f3579-b836-485b-923c-21f2ba37336a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:01.921062 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:01.921040 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsvh7\" (UniqueName: \"kubernetes.io/projected/4c8f3579-b836-485b-923c-21f2ba37336a-kube-api-access-vsvh7\") pod \"maas-default-gateway-openshift-default-78bb78ff78-x7llj\" (UID: \"4c8f3579-b836-485b-923c-21f2ba37336a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:02.074153 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:02.074071 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:02.194275 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:02.194248 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:02.218649 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:02.218626 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj"] Aug 05 16:31:02.220784 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:31:02.220757 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c8f3579_b836_485b_923c_21f2ba37336a.slice/crio-fe7d57082f7f4af28284897b9545996c18a7449f7591139517b325ceeed9b6c1 WatchSource:0}: Error finding container fe7d57082f7f4af28284897b9545996c18a7449f7591139517b325ceeed9b6c1: Status 404 returned error can't find the container with id fe7d57082f7f4af28284897b9545996c18a7449f7591139517b325ceeed9b6c1 Aug 05 16:31:02.222669 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:02.222642 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:31:02.222764 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:02.222701 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:31:02.222764 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:02.222729 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:31:02.435950 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:02.435840 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" event={"ID":"4c8f3579-b836-485b-923c-21f2ba37336a","Type":"ContainerStarted","Data":"87fa1564a4359c8df1c98370ccdeec531111dae38bac08641eebec987be51382"} Aug 05 16:31:02.435950 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:02.435933 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" event={"ID":"4c8f3579-b836-485b-923c-21f2ba37336a","Type":"ContainerStarted","Data":"fe7d57082f7f4af28284897b9545996c18a7449f7591139517b325ceeed9b6c1"} Aug 05 16:31:02.458702 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:02.458649 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" podStartSLOduration=1.458632973 podStartE2EDuration="1.458632973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:31:02.457623184 +0000 UTC m=+577.614469086" watchObservedRunningTime="2026-08-05 16:31:02.458632973 +0000 UTC m=+577.615478942" Aug 05 16:31:03.075018 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:03.074982 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:03.079458 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:03.079436 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:03.440733 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:03.440652 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:03.441510 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:03.441492 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:31:03.560040 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:03.560004 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:03.676904 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:03.676867 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:03.771258 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:03.771231 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:04.437954 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:04.437915 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-b8hzb"] Aug 05 16:31:04.440946 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:04.440921 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-b8hzb" Aug 05 16:31:04.444554 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:04.444530 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pjs79\"" Aug 05 16:31:04.464943 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:04.464918 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-b8hzb"] Aug 05 16:31:04.535927 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:04.535894 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv5bj\" (UniqueName: \"kubernetes.io/projected/fa6a4a8f-1b08-4bd9-a294-0b3883aae8e6-kube-api-access-dv5bj\") pod \"authorino-b54df94b4-b8hzb\" (UID: \"fa6a4a8f-1b08-4bd9-a294-0b3883aae8e6\") " pod="kuadrant-system/authorino-b54df94b4-b8hzb" Aug 05 16:31:04.537183 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:04.537159 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:04.636600 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:04.636566 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv5bj\" (UniqueName: \"kubernetes.io/projected/fa6a4a8f-1b08-4bd9-a294-0b3883aae8e6-kube-api-access-dv5bj\") pod \"authorino-b54df94b4-b8hzb\" (UID: \"fa6a4a8f-1b08-4bd9-a294-0b3883aae8e6\") " pod="kuadrant-system/authorino-b54df94b4-b8hzb" Aug 05 16:31:04.651327 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:04.651295 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:04.652197 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:04.652177 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv5bj\" (UniqueName: \"kubernetes.io/projected/fa6a4a8f-1b08-4bd9-a294-0b3883aae8e6-kube-api-access-dv5bj\") pod \"authorino-b54df94b4-b8hzb\" (UID: \"fa6a4a8f-1b08-4bd9-a294-0b3883aae8e6\") " pod="kuadrant-system/authorino-b54df94b4-b8hzb" Aug 05 16:31:04.696267 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:04.696200 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-g5hbm"] Aug 05 16:31:04.698220 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:04.698205 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-g5hbm" Aug 05 16:31:04.711249 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:04.711226 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-g5hbm"] Aug 05 16:31:04.736957 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:04.736930 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbtbd\" (UniqueName: \"kubernetes.io/projected/5534abd3-1176-44ee-8d9e-dd369f9d2833-kube-api-access-pbtbd\") pod \"authorino-c548b5d7d-g5hbm\" (UID: \"5534abd3-1176-44ee-8d9e-dd369f9d2833\") " pod="kuadrant-system/authorino-c548b5d7d-g5hbm" Aug 05 16:31:04.751875 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:04.751853 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-b8hzb" Aug 05 16:31:04.838087 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:04.838049 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbtbd\" (UniqueName: \"kubernetes.io/projected/5534abd3-1176-44ee-8d9e-dd369f9d2833-kube-api-access-pbtbd\") pod \"authorino-c548b5d7d-g5hbm\" (UID: \"5534abd3-1176-44ee-8d9e-dd369f9d2833\") " pod="kuadrant-system/authorino-c548b5d7d-g5hbm" Aug 05 16:31:04.848435 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:04.848410 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbtbd\" (UniqueName: \"kubernetes.io/projected/5534abd3-1176-44ee-8d9e-dd369f9d2833-kube-api-access-pbtbd\") pod \"authorino-c548b5d7d-g5hbm\" (UID: \"5534abd3-1176-44ee-8d9e-dd369f9d2833\") " pod="kuadrant-system/authorino-c548b5d7d-g5hbm" Aug 05 16:31:04.883278 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:04.883253 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-b8hzb"] Aug 05 16:31:04.885840 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:31:04.885811 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa6a4a8f_1b08_4bd9_a294_0b3883aae8e6.slice/crio-b9a946c31d8541bc23af1ab21647d8eb3f746ba7877a5667b6f4707cab6f57e0 WatchSource:0}: Error finding container b9a946c31d8541bc23af1ab21647d8eb3f746ba7877a5667b6f4707cab6f57e0: Status 404 returned error can't find the container with id b9a946c31d8541bc23af1ab21647d8eb3f746ba7877a5667b6f4707cab6f57e0 Aug 05 16:31:05.006651 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:05.006564 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-g5hbm" Aug 05 16:31:05.131795 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:05.131768 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-g5hbm"] Aug 05 16:31:05.133910 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:31:05.133880 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5534abd3_1176_44ee_8d9e_dd369f9d2833.slice/crio-a7a7bcdb888b1475500c441c1f9425f377b4938ab98e4aa38f21aa33f4f0fb6d WatchSource:0}: Error finding container a7a7bcdb888b1475500c441c1f9425f377b4938ab98e4aa38f21aa33f4f0fb6d: Status 404 returned error can't find the container with id a7a7bcdb888b1475500c441c1f9425f377b4938ab98e4aa38f21aa33f4f0fb6d Aug 05 16:31:05.451709 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:05.451663 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-b8hzb" event={"ID":"fa6a4a8f-1b08-4bd9-a294-0b3883aae8e6","Type":"ContainerStarted","Data":"b9a946c31d8541bc23af1ab21647d8eb3f746ba7877a5667b6f4707cab6f57e0"} Aug 05 16:31:05.452913 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:05.452882 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-g5hbm" event={"ID":"5534abd3-1176-44ee-8d9e-dd369f9d2833","Type":"ContainerStarted","Data":"a7a7bcdb888b1475500c441c1f9425f377b4938ab98e4aa38f21aa33f4f0fb6d"} Aug 05 16:31:08.468073 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:08.468033 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-g5hbm" event={"ID":"5534abd3-1176-44ee-8d9e-dd369f9d2833","Type":"ContainerStarted","Data":"a9886df262e2b4bfce5056dfae8b7995794bfb5053778990e4544817ec02acec"} Aug 05 16:31:08.469568 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:08.469541 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-b8hzb" event={"ID":"fa6a4a8f-1b08-4bd9-a294-0b3883aae8e6","Type":"ContainerStarted","Data":"8e043c8017209eee243e72fc93934bb860683ebe5849e7a3caca0592752fcaf8"} Aug 05 16:31:08.542362 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:08.542298 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-b8hzb" podStartSLOduration=1.583792505 podStartE2EDuration="4.542278044s" totalImagesPullingTime="2.958485539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:31:08.535223132 +0000 UTC m=+583.692069034" watchObservedRunningTime="2026-08-05 16:31:08.542278044 +0000 UTC m=+583.699123947" Aug 05 16:31:08.543666 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:08.543631 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-g5hbm" podStartSLOduration=1.8231230649999999 podStartE2EDuration="4.543619376s" totalImagesPullingTime="2.720496311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:31:08.50073872 +0000 UTC m=+583.657584621" watchObservedRunningTime="2026-08-05 16:31:08.543619376 +0000 UTC m=+583.700465291" Aug 05 16:31:08.551433 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:08.551408 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-b8hzb"] Aug 05 16:31:10.477307 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:10.477271 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-b8hzb" podUID="fa6a4a8f-1b08-4bd9-a294-0b3883aae8e6" containerName="authorino" containerID="cri-o://8e043c8017209eee243e72fc93934bb860683ebe5849e7a3caca0592752fcaf8" gracePeriod=30 Aug 05 16:31:10.712139 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:10.712113 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-b8hzb" Aug 05 16:31:10.788433 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:10.788408 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dv5bj\" (UniqueName: \"kubernetes.io/projected/fa6a4a8f-1b08-4bd9-a294-0b3883aae8e6-kube-api-access-dv5bj\") pod \"fa6a4a8f-1b08-4bd9-a294-0b3883aae8e6\" (UID: \"fa6a4a8f-1b08-4bd9-a294-0b3883aae8e6\") " Aug 05 16:31:10.790403 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:10.790364 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa6a4a8f-1b08-4bd9-a294-0b3883aae8e6-kube-api-access-dv5bj" (OuterVolumeSpecName: "kube-api-access-dv5bj") pod "fa6a4a8f-1b08-4bd9-a294-0b3883aae8e6" (UID: "fa6a4a8f-1b08-4bd9-a294-0b3883aae8e6"). InnerVolumeSpecName "kube-api-access-dv5bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:31:10.889573 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:10.889540 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv5bj\" (UniqueName: \"kubernetes.io/projected/fa6a4a8f-1b08-4bd9-a294-0b3883aae8e6-kube-api-access-dv5bj\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:31:10.897583 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:10.897557 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:11.482998 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:11.482968 2573 generic.go:356] "Generic (PLEG): container finished" podID="fa6a4a8f-1b08-4bd9-a294-0b3883aae8e6" containerID="8e043c8017209eee243e72fc93934bb860683ebe5849e7a3caca0592752fcaf8" exitCode=0 Aug 05 16:31:11.483423 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:11.483021 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-b8hzb" Aug 05 16:31:11.483423 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:11.483057 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-b8hzb" event={"ID":"fa6a4a8f-1b08-4bd9-a294-0b3883aae8e6","Type":"ContainerDied","Data":"8e043c8017209eee243e72fc93934bb860683ebe5849e7a3caca0592752fcaf8"} Aug 05 16:31:11.483423 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:11.483100 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-b8hzb" event={"ID":"fa6a4a8f-1b08-4bd9-a294-0b3883aae8e6","Type":"ContainerDied","Data":"b9a946c31d8541bc23af1ab21647d8eb3f746ba7877a5667b6f4707cab6f57e0"} Aug 05 16:31:11.483423 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:11.483122 2573 scope.go:117] "RemoveContainer" containerID="8e043c8017209eee243e72fc93934bb860683ebe5849e7a3caca0592752fcaf8" Aug 05 16:31:11.490942 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:11.490923 2573 scope.go:117] "RemoveContainer" containerID="8e043c8017209eee243e72fc93934bb860683ebe5849e7a3caca0592752fcaf8" Aug 05 16:31:11.491190 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:31:11.491173 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e043c8017209eee243e72fc93934bb860683ebe5849e7a3caca0592752fcaf8\": container with ID starting with 8e043c8017209eee243e72fc93934bb860683ebe5849e7a3caca0592752fcaf8 not found: ID does not exist" containerID="8e043c8017209eee243e72fc93934bb860683ebe5849e7a3caca0592752fcaf8" Aug 05 16:31:11.491234 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:11.491198 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e043c8017209eee243e72fc93934bb860683ebe5849e7a3caca0592752fcaf8"} err="failed to get container status \"8e043c8017209eee243e72fc93934bb860683ebe5849e7a3caca0592752fcaf8\": rpc error: code = NotFound desc = could not find container \"8e043c8017209eee243e72fc93934bb860683ebe5849e7a3caca0592752fcaf8\": container with ID starting with 8e043c8017209eee243e72fc93934bb860683ebe5849e7a3caca0592752fcaf8 not found: ID does not exist" Aug 05 16:31:11.503843 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:11.503817 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-b8hzb"] Aug 05 16:31:11.508665 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:11.508646 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-b8hzb"] Aug 05 16:31:13.303645 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:13.303616 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa6a4a8f-1b08-4bd9-a294-0b3883aae8e6" path="/var/lib/kubelet/pods/fa6a4a8f-1b08-4bd9-a294-0b3883aae8e6/volumes" Aug 05 16:31:19.648345 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:19.648307 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:19.754250 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:19.754201 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:20.138971 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:20.138915 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:32.310904 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:32.310852 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:32.409975 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:32.409917 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:32.520283 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:32.520252 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-4k9kn"] Aug 05 16:31:32.520568 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:32.520556 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa6a4a8f-1b08-4bd9-a294-0b3883aae8e6" containerName="authorino" Aug 05 16:31:32.520615 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:32.520569 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa6a4a8f-1b08-4bd9-a294-0b3883aae8e6" containerName="authorino" Aug 05 16:31:32.520654 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:32.520633 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa6a4a8f-1b08-4bd9-a294-0b3883aae8e6" containerName="authorino" Aug 05 16:31:32.524779 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:32.524760 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-4k9kn" Aug 05 16:31:32.534104 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:32.534083 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-4k9kn"] Aug 05 16:31:32.593405 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:32.593321 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-4k9kn"] Aug 05 16:31:32.593623 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:31:32.593602 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-q8g4r], unattached volumes=[], failed to process volumes=[kube-api-access-q8g4r]: context canceled" pod="kuadrant-system/authorino-5dc7ccb5f-4k9kn" podUID="e9d5a0d9-26d9-4738-8b26-97d83899a431" Aug 05 16:31:32.624675 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:32.624644 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687bd45585-qkldn"] Aug 05 16:31:32.633035 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:32.633011 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-qkldn" Aug 05 16:31:32.636646 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:32.636627 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Aug 05 16:31:32.637990 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:32.637967 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687bd45585-qkldn"] Aug 05 16:31:32.677809 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:32.677781 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8g4r\" (UniqueName: \"kubernetes.io/projected/e9d5a0d9-26d9-4738-8b26-97d83899a431-kube-api-access-q8g4r\") pod \"authorino-5dc7ccb5f-4k9kn\" (UID: \"e9d5a0d9-26d9-4738-8b26-97d83899a431\") " pod="kuadrant-system/authorino-5dc7ccb5f-4k9kn" Aug 05 16:31:32.774193 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:32.774159 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-qkldn"] Aug 05 16:31:32.774425 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:31:32.774404 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-vg2wx tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-687bd45585-qkldn" podUID="3ac7e491-4ca4-4bb6-97d5-33b70945276d" Aug 05 16:31:32.778977 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:32.778952 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg2wx\" (UniqueName: \"kubernetes.io/projected/3ac7e491-4ca4-4bb6-97d5-33b70945276d-kube-api-access-vg2wx\") pod \"authorino-687bd45585-qkldn\" (UID: \"3ac7e491-4ca4-4bb6-97d5-33b70945276d\") " pod="kuadrant-system/authorino-687bd45585-qkldn" Aug 05 16:31:32.779039 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:32.778991 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3ac7e491-4ca4-4bb6-97d5-33b70945276d-tls-cert\") pod \"authorino-687bd45585-qkldn\" (UID: \"3ac7e491-4ca4-4bb6-97d5-33b70945276d\") " pod="kuadrant-system/authorino-687bd45585-qkldn" Aug 05 16:31:32.779039 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:32.779031 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8g4r\" (UniqueName: \"kubernetes.io/projected/e9d5a0d9-26d9-4738-8b26-97d83899a431-kube-api-access-q8g4r\") pod \"authorino-5dc7ccb5f-4k9kn\" (UID: \"e9d5a0d9-26d9-4738-8b26-97d83899a431\") " pod="kuadrant-system/authorino-5dc7ccb5f-4k9kn" Aug 05 16:31:32.792722 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:32.792686 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8g4r\" (UniqueName: \"kubernetes.io/projected/e9d5a0d9-26d9-4738-8b26-97d83899a431-kube-api-access-q8g4r\") pod \"authorino-5dc7ccb5f-4k9kn\" (UID: \"e9d5a0d9-26d9-4738-8b26-97d83899a431\") " pod="kuadrant-system/authorino-5dc7ccb5f-4k9kn" Aug 05 16:31:32.815411 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:32.815385 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7954cf84df-5hxz9"] Aug 05 16:31:32.818730 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:32.818714 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7954cf84df-5hxz9" Aug 05 16:31:32.833535 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:32.833513 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7954cf84df-5hxz9"] Aug 05 16:31:32.880075 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:32.880008 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg2wx\" (UniqueName: \"kubernetes.io/projected/3ac7e491-4ca4-4bb6-97d5-33b70945276d-kube-api-access-vg2wx\") pod \"authorino-687bd45585-qkldn\" (UID: \"3ac7e491-4ca4-4bb6-97d5-33b70945276d\") " pod="kuadrant-system/authorino-687bd45585-qkldn" Aug 05 16:31:32.880075 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:32.880046 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3ac7e491-4ca4-4bb6-97d5-33b70945276d-tls-cert\") pod \"authorino-687bd45585-qkldn\" (UID: \"3ac7e491-4ca4-4bb6-97d5-33b70945276d\") " pod="kuadrant-system/authorino-687bd45585-qkldn" Aug 05 16:31:32.882604 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:32.882583 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3ac7e491-4ca4-4bb6-97d5-33b70945276d-tls-cert\") pod \"authorino-687bd45585-qkldn\" (UID: \"3ac7e491-4ca4-4bb6-97d5-33b70945276d\") " pod="kuadrant-system/authorino-687bd45585-qkldn" Aug 05 16:31:32.890107 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:32.890086 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg2wx\" (UniqueName: \"kubernetes.io/projected/3ac7e491-4ca4-4bb6-97d5-33b70945276d-kube-api-access-vg2wx\") pod \"authorino-687bd45585-qkldn\" (UID: \"3ac7e491-4ca4-4bb6-97d5-33b70945276d\") " pod="kuadrant-system/authorino-687bd45585-qkldn" Aug 05 16:31:32.981058 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:32.981017 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/71aa4956-b041-45be-92f7-08905c24c48e-tls-cert\") pod \"authorino-7954cf84df-5hxz9\" (UID: \"71aa4956-b041-45be-92f7-08905c24c48e\") " pod="kuadrant-system/authorino-7954cf84df-5hxz9" Aug 05 16:31:32.981221 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:32.981067 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrbcz\" (UniqueName: \"kubernetes.io/projected/71aa4956-b041-45be-92f7-08905c24c48e-kube-api-access-qrbcz\") pod \"authorino-7954cf84df-5hxz9\" (UID: \"71aa4956-b041-45be-92f7-08905c24c48e\") " pod="kuadrant-system/authorino-7954cf84df-5hxz9" Aug 05 16:31:33.082447 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:33.082411 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/71aa4956-b041-45be-92f7-08905c24c48e-tls-cert\") pod \"authorino-7954cf84df-5hxz9\" (UID: \"71aa4956-b041-45be-92f7-08905c24c48e\") " pod="kuadrant-system/authorino-7954cf84df-5hxz9" Aug 05 16:31:33.082634 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:33.082454 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrbcz\" (UniqueName: \"kubernetes.io/projected/71aa4956-b041-45be-92f7-08905c24c48e-kube-api-access-qrbcz\") pod \"authorino-7954cf84df-5hxz9\" (UID: \"71aa4956-b041-45be-92f7-08905c24c48e\") " pod="kuadrant-system/authorino-7954cf84df-5hxz9" Aug 05 16:31:33.084844 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:33.084820 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/71aa4956-b041-45be-92f7-08905c24c48e-tls-cert\") pod \"authorino-7954cf84df-5hxz9\" (UID: \"71aa4956-b041-45be-92f7-08905c24c48e\") " pod="kuadrant-system/authorino-7954cf84df-5hxz9" Aug 05 16:31:33.093461 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:33.093430 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrbcz\" (UniqueName: \"kubernetes.io/projected/71aa4956-b041-45be-92f7-08905c24c48e-kube-api-access-qrbcz\") pod \"authorino-7954cf84df-5hxz9\" (UID: \"71aa4956-b041-45be-92f7-08905c24c48e\") " pod="kuadrant-system/authorino-7954cf84df-5hxz9" Aug 05 16:31:33.127369 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:33.127344 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7954cf84df-5hxz9" Aug 05 16:31:33.253942 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:33.253911 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7954cf84df-5hxz9"] Aug 05 16:31:33.256770 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:31:33.256744 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71aa4956_b041_45be_92f7_08905c24c48e.slice/crio-4f927fd4dbed1206a2bd2409ba6227327e9985f30f04573fce093614cfbd55b3 WatchSource:0}: Error finding container 4f927fd4dbed1206a2bd2409ba6227327e9985f30f04573fce093614cfbd55b3: Status 404 returned error can't find the container with id 4f927fd4dbed1206a2bd2409ba6227327e9985f30f04573fce093614cfbd55b3 Aug 05 16:31:33.582063 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:33.582034 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-qkldn" Aug 05 16:31:33.582063 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:33.582052 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7954cf84df-5hxz9" event={"ID":"71aa4956-b041-45be-92f7-08905c24c48e","Type":"ContainerStarted","Data":"4f927fd4dbed1206a2bd2409ba6227327e9985f30f04573fce093614cfbd55b3"} Aug 05 16:31:33.582569 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:33.582180 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-4k9kn" Aug 05 16:31:33.590050 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:33.590031 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-qkldn" Aug 05 16:31:33.592064 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:33.592051 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-4k9kn" Aug 05 16:31:33.685526 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:33.685495 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vg2wx\" (UniqueName: \"kubernetes.io/projected/3ac7e491-4ca4-4bb6-97d5-33b70945276d-kube-api-access-vg2wx\") pod \"3ac7e491-4ca4-4bb6-97d5-33b70945276d\" (UID: \"3ac7e491-4ca4-4bb6-97d5-33b70945276d\") " Aug 05 16:31:33.685526 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:33.685524 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3ac7e491-4ca4-4bb6-97d5-33b70945276d-tls-cert\") pod \"3ac7e491-4ca4-4bb6-97d5-33b70945276d\" (UID: \"3ac7e491-4ca4-4bb6-97d5-33b70945276d\") " Aug 05 16:31:33.685761 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:33.685552 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8g4r\" (UniqueName: \"kubernetes.io/projected/e9d5a0d9-26d9-4738-8b26-97d83899a431-kube-api-access-q8g4r\") pod \"e9d5a0d9-26d9-4738-8b26-97d83899a431\" (UID: \"e9d5a0d9-26d9-4738-8b26-97d83899a431\") " Aug 05 16:31:33.687764 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:33.687732 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac7e491-4ca4-4bb6-97d5-33b70945276d-kube-api-access-vg2wx" (OuterVolumeSpecName: "kube-api-access-vg2wx") pod "3ac7e491-4ca4-4bb6-97d5-33b70945276d" (UID: "3ac7e491-4ca4-4bb6-97d5-33b70945276d"). InnerVolumeSpecName "kube-api-access-vg2wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:31:33.687874 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:33.687795 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d5a0d9-26d9-4738-8b26-97d83899a431-kube-api-access-q8g4r" (OuterVolumeSpecName: "kube-api-access-q8g4r") pod "e9d5a0d9-26d9-4738-8b26-97d83899a431" (UID: "e9d5a0d9-26d9-4738-8b26-97d83899a431"). InnerVolumeSpecName "kube-api-access-q8g4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:31:33.687874 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:33.687805 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ac7e491-4ca4-4bb6-97d5-33b70945276d-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "3ac7e491-4ca4-4bb6-97d5-33b70945276d" (UID: "3ac7e491-4ca4-4bb6-97d5-33b70945276d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 16:31:33.786071 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:33.786039 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg2wx\" (UniqueName: \"kubernetes.io/projected/3ac7e491-4ca4-4bb6-97d5-33b70945276d-kube-api-access-vg2wx\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:31:33.786071 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:33.786065 2573 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3ac7e491-4ca4-4bb6-97d5-33b70945276d-tls-cert\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:31:33.786071 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:33.786074 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8g4r\" (UniqueName: \"kubernetes.io/projected/e9d5a0d9-26d9-4738-8b26-97d83899a431-kube-api-access-q8g4r\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:31:34.587789 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:34.587761 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-4k9kn" Aug 05 16:31:34.587789 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:34.587764 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7954cf84df-5hxz9" event={"ID":"71aa4956-b041-45be-92f7-08905c24c48e","Type":"ContainerStarted","Data":"35e2113755a76794466c4cb90363074f0ad388eb1acfe3b94fadb8fa408ceafb"} Aug 05 16:31:34.588219 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:34.587949 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-qkldn" Aug 05 16:31:34.605545 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:34.605500 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7954cf84df-5hxz9" podStartSLOduration=1.79531208 podStartE2EDuration="2.605485364s" totalImagesPullingTime="810.173284ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:31:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:31:34.605251868 +0000 UTC m=+609.762097773" watchObservedRunningTime="2026-08-05 16:31:34.605485364 +0000 UTC m=+609.762331294" Aug 05 16:31:34.633336 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:34.633308 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-4k9kn"] Aug 05 16:31:34.641190 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:34.641166 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-g5hbm"] Aug 05 16:31:34.641406 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:34.641336 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-g5hbm" podUID="5534abd3-1176-44ee-8d9e-dd369f9d2833" containerName="authorino" containerID="cri-o://a9886df262e2b4bfce5056dfae8b7995794bfb5053778990e4544817ec02acec" gracePeriod=30 Aug 05 16:31:34.647613 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:34.647590 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-4k9kn"] Aug 05 16:31:34.699037 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:34.699007 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-qkldn"] Aug 05 16:31:34.700641 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:34.700619 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-687bd45585-qkldn"] Aug 05 16:31:34.887236 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:34.887208 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-g5hbm" Aug 05 16:31:34.997369 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:34.997338 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbtbd\" (UniqueName: \"kubernetes.io/projected/5534abd3-1176-44ee-8d9e-dd369f9d2833-kube-api-access-pbtbd\") pod \"5534abd3-1176-44ee-8d9e-dd369f9d2833\" (UID: \"5534abd3-1176-44ee-8d9e-dd369f9d2833\") " Aug 05 16:31:34.999432 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:34.999409 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5534abd3-1176-44ee-8d9e-dd369f9d2833-kube-api-access-pbtbd" (OuterVolumeSpecName: "kube-api-access-pbtbd") pod "5534abd3-1176-44ee-8d9e-dd369f9d2833" (UID: "5534abd3-1176-44ee-8d9e-dd369f9d2833"). InnerVolumeSpecName "kube-api-access-pbtbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:31:35.098128 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:35.098093 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbtbd\" (UniqueName: \"kubernetes.io/projected/5534abd3-1176-44ee-8d9e-dd369f9d2833-kube-api-access-pbtbd\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:31:35.303655 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:35.303620 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ac7e491-4ca4-4bb6-97d5-33b70945276d" path="/var/lib/kubelet/pods/3ac7e491-4ca4-4bb6-97d5-33b70945276d/volumes" Aug 05 16:31:35.303832 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:35.303820 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9d5a0d9-26d9-4738-8b26-97d83899a431" path="/var/lib/kubelet/pods/e9d5a0d9-26d9-4738-8b26-97d83899a431/volumes" Aug 05 16:31:35.593174 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:35.593084 2573 generic.go:356] "Generic (PLEG): container finished" podID="5534abd3-1176-44ee-8d9e-dd369f9d2833" containerID="a9886df262e2b4bfce5056dfae8b7995794bfb5053778990e4544817ec02acec" exitCode=0 Aug 05 16:31:35.593174 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:35.593124 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-g5hbm" Aug 05 16:31:35.593691 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:35.593171 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-g5hbm" event={"ID":"5534abd3-1176-44ee-8d9e-dd369f9d2833","Type":"ContainerDied","Data":"a9886df262e2b4bfce5056dfae8b7995794bfb5053778990e4544817ec02acec"} Aug 05 16:31:35.593691 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:35.593213 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-g5hbm" event={"ID":"5534abd3-1176-44ee-8d9e-dd369f9d2833","Type":"ContainerDied","Data":"a7a7bcdb888b1475500c441c1f9425f377b4938ab98e4aa38f21aa33f4f0fb6d"} Aug 05 16:31:35.593691 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:35.593231 2573 scope.go:117] "RemoveContainer" containerID="a9886df262e2b4bfce5056dfae8b7995794bfb5053778990e4544817ec02acec" Aug 05 16:31:35.600910 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:35.600896 2573 scope.go:117] "RemoveContainer" containerID="a9886df262e2b4bfce5056dfae8b7995794bfb5053778990e4544817ec02acec" Aug 05 16:31:35.601161 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:31:35.601140 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9886df262e2b4bfce5056dfae8b7995794bfb5053778990e4544817ec02acec\": container with ID starting with a9886df262e2b4bfce5056dfae8b7995794bfb5053778990e4544817ec02acec not found: ID does not exist" containerID="a9886df262e2b4bfce5056dfae8b7995794bfb5053778990e4544817ec02acec" Aug 05 16:31:35.601258 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:35.601176 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9886df262e2b4bfce5056dfae8b7995794bfb5053778990e4544817ec02acec"} err="failed to get container status \"a9886df262e2b4bfce5056dfae8b7995794bfb5053778990e4544817ec02acec\": rpc error: code = NotFound desc = could not find container \"a9886df262e2b4bfce5056dfae8b7995794bfb5053778990e4544817ec02acec\": container with ID starting with a9886df262e2b4bfce5056dfae8b7995794bfb5053778990e4544817ec02acec not found: ID does not exist" Aug 05 16:31:35.616345 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:35.616314 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-g5hbm"] Aug 05 16:31:35.624986 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:35.624967 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-g5hbm"] Aug 05 16:31:37.303917 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:37.303886 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5534abd3-1176-44ee-8d9e-dd369f9d2833" path="/var/lib/kubelet/pods/5534abd3-1176-44ee-8d9e-dd369f9d2833/volumes" Aug 05 16:31:39.438825 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:39.438784 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:39.597396 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:39.597352 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:39.696634 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:39.696534 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:42.719700 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:42.719667 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:42.823255 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:42.823223 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:31:54.410963 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:31:54.410882 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:14.479702 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:14.479672 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:16.802552 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:16.802508 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:16.996831 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:16.996793 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:17.228227 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:17.228142 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:19.648084 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:19.648050 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:19.741086 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:19.741054 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:26.209106 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:26.209079 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:26.307508 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:26.307476 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:26.805083 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:26.805050 2573 generic.go:356] "Generic (PLEG): container finished" podID="4c8f3579-b836-485b-923c-21f2ba37336a" containerID="87fa1564a4359c8df1c98370ccdeec531111dae38bac08641eebec987be51382" exitCode=0 Aug 05 16:32:26.805261 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:26.805095 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" event={"ID":"4c8f3579-b836-485b-923c-21f2ba37336a","Type":"ContainerDied","Data":"87fa1564a4359c8df1c98370ccdeec531111dae38bac08641eebec987be51382"} Aug 05 16:32:26.805503 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:26.805488 2573 scope.go:117] "RemoveContainer" containerID="87fa1564a4359c8df1c98370ccdeec531111dae38bac08641eebec987be51382" Aug 05 16:32:26.806533 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:26.806501 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:32:26.806654 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:26.806568 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:32:26.806654 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:26.806599 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:32:27.816524 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:27.816486 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" event={"ID":"4c8f3579-b836-485b-923c-21f2ba37336a","Type":"ContainerStarted","Data":"b465ca66286c63ddb1e25694bc747d98b1ce8af002c1634e0a28a7b3d6d43e29"} Aug 05 16:32:28.075162 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:28.075065 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:32:28.075319 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:28.075247 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-x7llj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Aug 05 16:32:28.075319 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:28.075293 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" podUID="4c8f3579-b836-485b-923c-21f2ba37336a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Aug 05 16:32:29.075117 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:29.075079 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-x7llj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Aug 05 16:32:29.075537 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:29.075135 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" podUID="4c8f3579-b836-485b-923c-21f2ba37336a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Aug 05 16:32:29.269692 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:29.269660 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:29.370567 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:29.370488 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:30.075525 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:30.075487 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-x7llj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Aug 05 16:32:30.075991 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:30.075552 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" podUID="4c8f3579-b836-485b-923c-21f2ba37336a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Aug 05 16:32:31.074953 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:31.074918 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-x7llj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Aug 05 16:32:31.075127 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:31.074975 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" podUID="4c8f3579-b836-485b-923c-21f2ba37336a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Aug 05 16:32:32.075416 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:32.075358 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-x7llj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Aug 05 16:32:32.075808 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:32.075439 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" podUID="4c8f3579-b836-485b-923c-21f2ba37336a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Aug 05 16:32:33.078942 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:33.078911 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:32:33.079405 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:33.079201 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:32:33.079865 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:33.079852 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-x7llj" Aug 05 16:32:34.885236 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:34.885201 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:36.595893 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:36.595859 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:32:39.334225 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:32:39.334188 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:33:08.943869 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:08.943833 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr"] Aug 05 16:33:08.944349 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:08.944150 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5534abd3-1176-44ee-8d9e-dd369f9d2833" containerName="authorino" Aug 05 16:33:08.944349 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:08.944162 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5534abd3-1176-44ee-8d9e-dd369f9d2833" containerName="authorino" Aug 05 16:33:08.944349 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:08.944212 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5534abd3-1176-44ee-8d9e-dd369f9d2833" containerName="authorino" Aug 05 16:33:08.949266 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:08.947485 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr" Aug 05 16:33:08.953359 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:08.953334 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Aug 05 16:33:08.953525 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:08.953334 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Aug 05 16:33:08.953525 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:08.953337 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dck4v\"" Aug 05 16:33:08.953640 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:08.953624 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Aug 05 16:33:08.960330 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:08.960306 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr"] Aug 05 16:33:08.968049 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:08.968029 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:33:09.075559 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:09.075525 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t9nfr\" (UID: \"bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr" Aug 05 16:33:09.075761 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:09.075570 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t9nfr\" (UID: \"bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr" Aug 05 16:33:09.075761 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:09.075592 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t9nfr\" (UID: \"bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr" Aug 05 16:33:09.075761 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:09.075732 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb468\" (UniqueName: \"kubernetes.io/projected/bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d-kube-api-access-cb468\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t9nfr\" (UID: \"bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr" Aug 05 16:33:09.075928 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:09.075804 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t9nfr\" (UID: \"bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr" Aug 05 16:33:09.075928 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:09.075845 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t9nfr\" (UID: \"bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr" Aug 05 16:33:09.176689 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:09.176654 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t9nfr\" (UID: \"bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr" Aug 05 16:33:09.176689 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:09.176695 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t9nfr\" (UID: \"bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr" Aug 05 16:33:09.176957 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:09.176751 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb468\" (UniqueName: \"kubernetes.io/projected/bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d-kube-api-access-cb468\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t9nfr\" (UID: \"bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr" Aug 05 16:33:09.176957 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:09.176806 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t9nfr\" (UID: \"bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr" Aug 05 16:33:09.176957 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:09.176835 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t9nfr\" (UID: \"bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr" Aug 05 16:33:09.176957 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:09.176864 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t9nfr\" (UID: \"bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr" Aug 05 16:33:09.177234 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:09.177211 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t9nfr\" (UID: \"bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr" Aug 05 16:33:09.177314 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:09.177250 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t9nfr\" (UID: \"bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr" Aug 05 16:33:09.177429 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:09.177347 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t9nfr\" (UID: \"bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr" Aug 05 16:33:09.179007 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:09.178987 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t9nfr\" (UID: \"bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr" Aug 05 16:33:09.179084 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:09.179011 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t9nfr\" (UID: \"bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr" Aug 05 16:33:09.187103 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:09.187084 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb468\" (UniqueName: \"kubernetes.io/projected/bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d-kube-api-access-cb468\") pod \"e2e-trlp-test-simulated-kserve-598466548d-t9nfr\" (UID: \"bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr" Aug 05 16:33:09.259849 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:09.259814 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr" Aug 05 16:33:09.405820 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:09.405788 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr"] Aug 05 16:33:09.408805 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:33:09.408776 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc6e8e0c_7870_4fe3_8e21_17d7e1798b0d.slice/crio-fd44c9eb7552c8d369622031de7fb6d4500a05673b1d952e37857a05753da977 WatchSource:0}: Error finding container fd44c9eb7552c8d369622031de7fb6d4500a05673b1d952e37857a05753da977: Status 404 returned error can't find the container with id fd44c9eb7552c8d369622031de7fb6d4500a05673b1d952e37857a05753da977 Aug 05 16:33:09.410018 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:09.410001 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 05 16:33:09.994155 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:09.994111 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr" event={"ID":"bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d","Type":"ContainerStarted","Data":"fd44c9eb7552c8d369622031de7fb6d4500a05673b1d952e37857a05753da977"} Aug 05 16:33:10.999459 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:10.999421 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr" event={"ID":"bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d","Type":"ContainerStarted","Data":"5c2d5804ad68cf8a7fcc3c85e8b33957ae729832803b93586a84e7fb3f021c2a"} Aug 05 16:33:11.024833 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:11.024785 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr" podStartSLOduration=1.611358392 podStartE2EDuration="3.024769874s" totalImagesPullingTime="1.413411482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:33:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:33:11.022462502 +0000 UTC m=+706.179308402" watchObservedRunningTime="2026-08-05 16:33:11.024769874 +0000 UTC m=+706.181615777" Aug 05 16:33:19.260202 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:19.260115 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr" Aug 05 16:33:19.260202 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:19.260184 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr" Aug 05 16:33:19.272758 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:19.272736 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr" Aug 05 16:33:20.048490 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:20.048464 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-t9nfr" Aug 05 16:33:23.285797 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:23.285755 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:33:50.039542 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:50.039502 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:33:59.583795 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:59.583747 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt"] Aug 05 16:33:59.587083 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:59.587066 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt" Aug 05 16:33:59.590508 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:59.590488 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Aug 05 16:33:59.597779 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:59.597756 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt"] Aug 05 16:33:59.716245 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:59.716211 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3cc148c-4a43-4867-b50a-60e042c781bf-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt\" (UID: \"a3cc148c-4a43-4867-b50a-60e042c781bf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt" Aug 05 16:33:59.716432 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:59.716257 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3cc148c-4a43-4867-b50a-60e042c781bf-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt\" (UID: \"a3cc148c-4a43-4867-b50a-60e042c781bf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt" Aug 05 16:33:59.716432 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:59.716333 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x2tg\" (UniqueName: \"kubernetes.io/projected/a3cc148c-4a43-4867-b50a-60e042c781bf-kube-api-access-5x2tg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt\" (UID: \"a3cc148c-4a43-4867-b50a-60e042c781bf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt" Aug 05 16:33:59.716432 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:59.716397 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3cc148c-4a43-4867-b50a-60e042c781bf-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt\" (UID: \"a3cc148c-4a43-4867-b50a-60e042c781bf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt" Aug 05 16:33:59.716432 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:59.716427 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3cc148c-4a43-4867-b50a-60e042c781bf-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt\" (UID: \"a3cc148c-4a43-4867-b50a-60e042c781bf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt" Aug 05 16:33:59.716589 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:59.716473 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3cc148c-4a43-4867-b50a-60e042c781bf-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt\" (UID: \"a3cc148c-4a43-4867-b50a-60e042c781bf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt" Aug 05 16:33:59.731367 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:59.731327 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:33:59.817613 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:59.817581 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3cc148c-4a43-4867-b50a-60e042c781bf-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt\" (UID: \"a3cc148c-4a43-4867-b50a-60e042c781bf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt" Aug 05 16:33:59.817821 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:59.817637 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x2tg\" (UniqueName: \"kubernetes.io/projected/a3cc148c-4a43-4867-b50a-60e042c781bf-kube-api-access-5x2tg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt\" (UID: \"a3cc148c-4a43-4867-b50a-60e042c781bf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt" Aug 05 16:33:59.817821 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:59.817667 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3cc148c-4a43-4867-b50a-60e042c781bf-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt\" (UID: \"a3cc148c-4a43-4867-b50a-60e042c781bf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt" Aug 05 16:33:59.817821 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:59.817708 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3cc148c-4a43-4867-b50a-60e042c781bf-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt\" (UID: \"a3cc148c-4a43-4867-b50a-60e042c781bf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt" Aug 05 16:33:59.817821 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:59.817747 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3cc148c-4a43-4867-b50a-60e042c781bf-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt\" (UID: \"a3cc148c-4a43-4867-b50a-60e042c781bf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt" Aug 05 16:33:59.817821 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:59.817783 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3cc148c-4a43-4867-b50a-60e042c781bf-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt\" (UID: \"a3cc148c-4a43-4867-b50a-60e042c781bf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt" Aug 05 16:33:59.818109 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:59.818006 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3cc148c-4a43-4867-b50a-60e042c781bf-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt\" (UID: \"a3cc148c-4a43-4867-b50a-60e042c781bf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt" Aug 05 16:33:59.818109 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:59.818059 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3cc148c-4a43-4867-b50a-60e042c781bf-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt\" (UID: \"a3cc148c-4a43-4867-b50a-60e042c781bf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt" Aug 05 16:33:59.818395 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:59.818345 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3cc148c-4a43-4867-b50a-60e042c781bf-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt\" (UID: \"a3cc148c-4a43-4867-b50a-60e042c781bf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt" Aug 05 16:33:59.820320 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:59.820297 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3cc148c-4a43-4867-b50a-60e042c781bf-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt\" (UID: \"a3cc148c-4a43-4867-b50a-60e042c781bf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt" Aug 05 16:33:59.820666 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:59.820648 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3cc148c-4a43-4867-b50a-60e042c781bf-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt\" (UID: \"a3cc148c-4a43-4867-b50a-60e042c781bf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt" Aug 05 16:33:59.838906 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:59.838853 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x2tg\" (UniqueName: \"kubernetes.io/projected/a3cc148c-4a43-4867-b50a-60e042c781bf-kube-api-access-5x2tg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt\" (UID: \"a3cc148c-4a43-4867-b50a-60e042c781bf\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt" Aug 05 16:33:59.895585 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:33:59.895539 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt" Aug 05 16:34:00.030138 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:00.030111 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt"] Aug 05 16:34:00.032576 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:34:00.032548 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3cc148c_4a43_4867_b50a_60e042c781bf.slice/crio-d0e8cc404ebc0076d43f381bb3f830a7691898e8ddef3670ccf78ede0d8ac9be WatchSource:0}: Error finding container d0e8cc404ebc0076d43f381bb3f830a7691898e8ddef3670ccf78ede0d8ac9be: Status 404 returned error can't find the container with id d0e8cc404ebc0076d43f381bb3f830a7691898e8ddef3670ccf78ede0d8ac9be Aug 05 16:34:00.217660 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:00.217581 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt" event={"ID":"a3cc148c-4a43-4867-b50a-60e042c781bf","Type":"ContainerStarted","Data":"d0e8cc404ebc0076d43f381bb3f830a7691898e8ddef3670ccf78ede0d8ac9be"} Aug 05 16:34:01.224116 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:01.224084 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt" event={"ID":"a3cc148c-4a43-4867-b50a-60e042c781bf","Type":"ContainerStarted","Data":"87ff961b91d07edeb8ad1c5d1677ff26ca2066d60d532d16d8ac1d27b1364f88"} Aug 05 16:34:01.243896 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:01.243848 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt" podStartSLOduration=1.9901464180000001 podStartE2EDuration="2.243832492s" totalImagesPullingTime="253.686074ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:34:01.242991894 +0000 UTC m=+756.399837795" watchObservedRunningTime="2026-08-05 16:34:01.243832492 +0000 UTC m=+756.400678608" Aug 05 16:34:09.895917 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:09.895881 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt" Aug 05 16:34:09.896309 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:09.895930 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt" Aug 05 16:34:09.908724 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:09.908696 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt" Aug 05 16:34:10.274503 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:10.274477 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt" Aug 05 16:34:52.345202 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:52.345113 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6cd787c69b-588vp"] Aug 05 16:34:52.348642 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:52.348621 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cd787c69b-588vp" Aug 05 16:34:52.359014 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:52.358986 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cd787c69b-588vp"] Aug 05 16:34:52.451779 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:52.451745 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8g5h\" (UniqueName: \"kubernetes.io/projected/7f30f2b9-98de-4d12-846e-c3511d31201c-kube-api-access-h8g5h\") pod \"authorino-6cd787c69b-588vp\" (UID: \"7f30f2b9-98de-4d12-846e-c3511d31201c\") " pod="kuadrant-system/authorino-6cd787c69b-588vp" Aug 05 16:34:52.451952 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:52.451799 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7f30f2b9-98de-4d12-846e-c3511d31201c-tls-cert\") pod \"authorino-6cd787c69b-588vp\" (UID: \"7f30f2b9-98de-4d12-846e-c3511d31201c\") " pod="kuadrant-system/authorino-6cd787c69b-588vp" Aug 05 16:34:52.553080 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:52.553044 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8g5h\" (UniqueName: \"kubernetes.io/projected/7f30f2b9-98de-4d12-846e-c3511d31201c-kube-api-access-h8g5h\") pod \"authorino-6cd787c69b-588vp\" (UID: \"7f30f2b9-98de-4d12-846e-c3511d31201c\") " pod="kuadrant-system/authorino-6cd787c69b-588vp" Aug 05 16:34:52.553255 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:52.553111 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7f30f2b9-98de-4d12-846e-c3511d31201c-tls-cert\") pod \"authorino-6cd787c69b-588vp\" (UID: \"7f30f2b9-98de-4d12-846e-c3511d31201c\") " pod="kuadrant-system/authorino-6cd787c69b-588vp" Aug 05 16:34:52.555666 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:52.555641 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7f30f2b9-98de-4d12-846e-c3511d31201c-tls-cert\") pod \"authorino-6cd787c69b-588vp\" (UID: \"7f30f2b9-98de-4d12-846e-c3511d31201c\") " pod="kuadrant-system/authorino-6cd787c69b-588vp" Aug 05 16:34:52.563267 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:52.563246 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8g5h\" (UniqueName: \"kubernetes.io/projected/7f30f2b9-98de-4d12-846e-c3511d31201c-kube-api-access-h8g5h\") pod \"authorino-6cd787c69b-588vp\" (UID: \"7f30f2b9-98de-4d12-846e-c3511d31201c\") " pod="kuadrant-system/authorino-6cd787c69b-588vp" Aug 05 16:34:52.657183 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:52.657091 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cd787c69b-588vp" Aug 05 16:34:52.785640 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:52.785605 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cd787c69b-588vp"] Aug 05 16:34:52.788741 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:34:52.788713 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f30f2b9_98de_4d12_846e_c3511d31201c.slice/crio-d72dcac89d30bb69d87e192b8769b6883c3f9840c7cf890a98b7b97c7985d4b5 WatchSource:0}: Error finding container d72dcac89d30bb69d87e192b8769b6883c3f9840c7cf890a98b7b97c7985d4b5: Status 404 returned error can't find the container with id d72dcac89d30bb69d87e192b8769b6883c3f9840c7cf890a98b7b97c7985d4b5 Aug 05 16:34:53.439502 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:53.439470 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cd787c69b-588vp" event={"ID":"7f30f2b9-98de-4d12-846e-c3511d31201c","Type":"ContainerStarted","Data":"d72dcac89d30bb69d87e192b8769b6883c3f9840c7cf890a98b7b97c7985d4b5"} Aug 05 16:34:54.448671 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:54.448626 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cd787c69b-588vp" event={"ID":"7f30f2b9-98de-4d12-846e-c3511d31201c","Type":"ContainerStarted","Data":"58be86b8f1a853937031208b023b04ead8c84ebb781fe7fbf772046719da8af5"} Aug 05 16:34:54.470398 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:54.470310 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6cd787c69b-588vp" podStartSLOduration=1.40661465 podStartE2EDuration="2.470289064s" totalImagesPullingTime="1.063674414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:34:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:34:54.470007303 +0000 UTC m=+809.626853205" watchObservedRunningTime="2026-08-05 16:34:54.470289064 +0000 UTC m=+809.627134967" Aug 05 16:34:54.501115 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:54.501078 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7954cf84df-5hxz9"] Aug 05 16:34:54.501346 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:54.501317 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7954cf84df-5hxz9" podUID="71aa4956-b041-45be-92f7-08905c24c48e" containerName="authorino" containerID="cri-o://35e2113755a76794466c4cb90363074f0ad388eb1acfe3b94fadb8fa408ceafb" gracePeriod=30 Aug 05 16:34:54.748152 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:54.748122 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7954cf84df-5hxz9" Aug 05 16:34:54.871547 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:54.871517 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrbcz\" (UniqueName: \"kubernetes.io/projected/71aa4956-b041-45be-92f7-08905c24c48e-kube-api-access-qrbcz\") pod \"71aa4956-b041-45be-92f7-08905c24c48e\" (UID: \"71aa4956-b041-45be-92f7-08905c24c48e\") " Aug 05 16:34:54.871705 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:54.871590 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/71aa4956-b041-45be-92f7-08905c24c48e-tls-cert\") pod \"71aa4956-b041-45be-92f7-08905c24c48e\" (UID: \"71aa4956-b041-45be-92f7-08905c24c48e\") " Aug 05 16:34:54.873687 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:54.873658 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71aa4956-b041-45be-92f7-08905c24c48e-kube-api-access-qrbcz" (OuterVolumeSpecName: "kube-api-access-qrbcz") pod "71aa4956-b041-45be-92f7-08905c24c48e" (UID: "71aa4956-b041-45be-92f7-08905c24c48e"). InnerVolumeSpecName "kube-api-access-qrbcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:34:54.882849 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:54.882816 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71aa4956-b041-45be-92f7-08905c24c48e-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "71aa4956-b041-45be-92f7-08905c24c48e" (UID: "71aa4956-b041-45be-92f7-08905c24c48e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 16:34:54.972513 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:54.972481 2573 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/71aa4956-b041-45be-92f7-08905c24c48e-tls-cert\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:34:54.972513 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:54.972511 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrbcz\" (UniqueName: \"kubernetes.io/projected/71aa4956-b041-45be-92f7-08905c24c48e-kube-api-access-qrbcz\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:34:55.018227 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:55.018193 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:34:55.136663 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:55.136611 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:34:55.454284 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:55.454253 2573 generic.go:356] "Generic (PLEG): container finished" podID="71aa4956-b041-45be-92f7-08905c24c48e" containerID="35e2113755a76794466c4cb90363074f0ad388eb1acfe3b94fadb8fa408ceafb" exitCode=0 Aug 05 16:34:55.454695 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:55.454300 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7954cf84df-5hxz9" Aug 05 16:34:55.454695 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:55.454345 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7954cf84df-5hxz9" event={"ID":"71aa4956-b041-45be-92f7-08905c24c48e","Type":"ContainerDied","Data":"35e2113755a76794466c4cb90363074f0ad388eb1acfe3b94fadb8fa408ceafb"} Aug 05 16:34:55.454695 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:55.454408 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7954cf84df-5hxz9" event={"ID":"71aa4956-b041-45be-92f7-08905c24c48e","Type":"ContainerDied","Data":"4f927fd4dbed1206a2bd2409ba6227327e9985f30f04573fce093614cfbd55b3"} Aug 05 16:34:55.454695 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:55.454435 2573 scope.go:117] "RemoveContainer" containerID="35e2113755a76794466c4cb90363074f0ad388eb1acfe3b94fadb8fa408ceafb" Aug 05 16:34:55.463542 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:55.463494 2573 scope.go:117] "RemoveContainer" containerID="35e2113755a76794466c4cb90363074f0ad388eb1acfe3b94fadb8fa408ceafb" Aug 05 16:34:55.463739 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:34:55.463718 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35e2113755a76794466c4cb90363074f0ad388eb1acfe3b94fadb8fa408ceafb\": container with ID starting with 35e2113755a76794466c4cb90363074f0ad388eb1acfe3b94fadb8fa408ceafb not found: ID does not exist" containerID="35e2113755a76794466c4cb90363074f0ad388eb1acfe3b94fadb8fa408ceafb" Aug 05 16:34:55.463806 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:55.463752 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e2113755a76794466c4cb90363074f0ad388eb1acfe3b94fadb8fa408ceafb"} err="failed to get container status \"35e2113755a76794466c4cb90363074f0ad388eb1acfe3b94fadb8fa408ceafb\": rpc error: code = NotFound desc = could not find container \"35e2113755a76794466c4cb90363074f0ad388eb1acfe3b94fadb8fa408ceafb\": container with ID starting with 35e2113755a76794466c4cb90363074f0ad388eb1acfe3b94fadb8fa408ceafb not found: ID does not exist" Aug 05 16:34:55.504581 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:55.504547 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7954cf84df-5hxz9"] Aug 05 16:34:55.512959 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:55.512936 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7954cf84df-5hxz9"] Aug 05 16:34:56.099978 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:56.099927 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:34:56.211325 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:56.211293 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:34:57.303519 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:34:57.303486 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71aa4956-b041-45be-92f7-08905c24c48e" path="/var/lib/kubelet/pods/71aa4956-b041-45be-92f7-08905c24c48e/volumes" Aug 05 16:37:00.089115 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:37:00.089081 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:37:01.229327 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:37:01.229295 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:37:10.880666 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:37:10.880629 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:37:10.974285 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:37:10.974248 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:38:01.847987 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:38:01.847909 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:38:03.540493 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:38:03.540462 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:38:04.554953 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:38:04.554915 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:38:04.893611 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:38:04.893520 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:38:08.528352 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:38:08.528316 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:38:08.604678 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:38:08.604651 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:38:08.723644 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:38:08.723604 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:38:08.907169 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:38:08.907086 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:38:15.018828 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:38:15.018795 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:38:15.115478 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:38:15.115449 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:39:43.047082 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:39:43.047046 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:41:09.988101 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:41:09.988013 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:42:04.360412 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:42:04.360363 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:42:04.462268 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:42:04.462236 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:43:37.569819 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:43:37.569779 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rw5c8"] Aug 05 16:43:37.570370 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:43:37.570036 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rw5c8" podUID="734db8c4-36a6-47ee-b61e-a5d77e7354b3" containerName="manager" containerID="cri-o://0dd26bfa89c4a5d7f803e17b706fe3d095ccddfc690598ba0e5f640e1a206e6e" gracePeriod=10 Aug 05 16:43:37.822294 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:43:37.822235 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rw5c8" Aug 05 16:43:37.923052 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:43:37.923013 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpcbt\" (UniqueName: \"kubernetes.io/projected/734db8c4-36a6-47ee-b61e-a5d77e7354b3-kube-api-access-gpcbt\") pod \"734db8c4-36a6-47ee-b61e-a5d77e7354b3\" (UID: \"734db8c4-36a6-47ee-b61e-a5d77e7354b3\") " Aug 05 16:43:37.923217 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:43:37.923093 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/734db8c4-36a6-47ee-b61e-a5d77e7354b3-extensions-socket-volume\") pod \"734db8c4-36a6-47ee-b61e-a5d77e7354b3\" (UID: \"734db8c4-36a6-47ee-b61e-a5d77e7354b3\") " Aug 05 16:43:37.923470 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:43:37.923436 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734db8c4-36a6-47ee-b61e-a5d77e7354b3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "734db8c4-36a6-47ee-b61e-a5d77e7354b3" (UID: "734db8c4-36a6-47ee-b61e-a5d77e7354b3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:43:37.925133 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:43:37.925106 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734db8c4-36a6-47ee-b61e-a5d77e7354b3-kube-api-access-gpcbt" (OuterVolumeSpecName: "kube-api-access-gpcbt") pod "734db8c4-36a6-47ee-b61e-a5d77e7354b3" (UID: "734db8c4-36a6-47ee-b61e-a5d77e7354b3"). InnerVolumeSpecName "kube-api-access-gpcbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:43:38.024390 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:43:38.024340 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpcbt\" (UniqueName: \"kubernetes.io/projected/734db8c4-36a6-47ee-b61e-a5d77e7354b3-kube-api-access-gpcbt\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:43:38.024390 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:43:38.024368 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/734db8c4-36a6-47ee-b61e-a5d77e7354b3-extensions-socket-volume\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:43:38.688335 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:43:38.688302 2573 generic.go:356] "Generic (PLEG): container finished" podID="734db8c4-36a6-47ee-b61e-a5d77e7354b3" containerID="0dd26bfa89c4a5d7f803e17b706fe3d095ccddfc690598ba0e5f640e1a206e6e" exitCode=0 Aug 05 16:43:38.688770 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:43:38.688361 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rw5c8" Aug 05 16:43:38.688770 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:43:38.688403 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rw5c8" event={"ID":"734db8c4-36a6-47ee-b61e-a5d77e7354b3","Type":"ContainerDied","Data":"0dd26bfa89c4a5d7f803e17b706fe3d095ccddfc690598ba0e5f640e1a206e6e"} Aug 05 16:43:38.688770 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:43:38.688453 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rw5c8" event={"ID":"734db8c4-36a6-47ee-b61e-a5d77e7354b3","Type":"ContainerDied","Data":"e39d1a7f3911b51f4c263a2e0b81b7c8ff7a13cf47f5721cee022b3ccd595201"} Aug 05 16:43:38.688770 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:43:38.688475 2573 scope.go:117] "RemoveContainer" containerID="0dd26bfa89c4a5d7f803e17b706fe3d095ccddfc690598ba0e5f640e1a206e6e" Aug 05 16:43:38.699442 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:43:38.699422 2573 scope.go:117] "RemoveContainer" containerID="0dd26bfa89c4a5d7f803e17b706fe3d095ccddfc690598ba0e5f640e1a206e6e" Aug 05 16:43:38.699685 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:43:38.699668 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dd26bfa89c4a5d7f803e17b706fe3d095ccddfc690598ba0e5f640e1a206e6e\": container with ID starting with 0dd26bfa89c4a5d7f803e17b706fe3d095ccddfc690598ba0e5f640e1a206e6e not found: ID does not exist" containerID="0dd26bfa89c4a5d7f803e17b706fe3d095ccddfc690598ba0e5f640e1a206e6e" Aug 05 16:43:38.699746 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:43:38.699694 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dd26bfa89c4a5d7f803e17b706fe3d095ccddfc690598ba0e5f640e1a206e6e"} err="failed to get container status \"0dd26bfa89c4a5d7f803e17b706fe3d095ccddfc690598ba0e5f640e1a206e6e\": rpc error: code = NotFound desc = could not find container \"0dd26bfa89c4a5d7f803e17b706fe3d095ccddfc690598ba0e5f640e1a206e6e\": container with ID starting with 0dd26bfa89c4a5d7f803e17b706fe3d095ccddfc690598ba0e5f640e1a206e6e not found: ID does not exist" Aug 05 16:43:38.716615 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:43:38.716593 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rw5c8"] Aug 05 16:43:38.721884 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:43:38.721860 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rw5c8"] Aug 05 16:43:39.304025 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:43:39.303995 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="734db8c4-36a6-47ee-b61e-a5d77e7354b3" path="/var/lib/kubelet/pods/734db8c4-36a6-47ee-b61e-a5d77e7354b3/volumes" Aug 05 16:44:38.246074 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:44:38.246039 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-l2cfn"] Aug 05 16:44:38.246497 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:44:38.246370 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71aa4956-b041-45be-92f7-08905c24c48e" containerName="authorino" Aug 05 16:44:38.246497 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:44:38.246404 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="71aa4956-b041-45be-92f7-08905c24c48e" containerName="authorino" Aug 05 16:44:38.246497 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:44:38.246433 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="734db8c4-36a6-47ee-b61e-a5d77e7354b3" containerName="manager" Aug 05 16:44:38.246497 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:44:38.246439 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="734db8c4-36a6-47ee-b61e-a5d77e7354b3" containerName="manager" Aug 05 16:44:38.246497 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:44:38.246483 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="734db8c4-36a6-47ee-b61e-a5d77e7354b3" containerName="manager" Aug 05 16:44:38.246497 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:44:38.246491 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="71aa4956-b041-45be-92f7-08905c24c48e" containerName="authorino" Aug 05 16:44:38.250267 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:44:38.250250 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-l2cfn" Aug 05 16:44:38.254283 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:44:38.254261 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vdhxn\"" Aug 05 16:44:38.269825 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:44:38.269792 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-l2cfn"] Aug 05 16:44:38.337311 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:44:38.337275 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3085d0d5-fb73-4f02-9dca-cdb3cf369470-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-l2cfn\" (UID: \"3085d0d5-fb73-4f02-9dca-cdb3cf369470\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-l2cfn" Aug 05 16:44:38.337502 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:44:38.337338 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkn7r\" (UniqueName: \"kubernetes.io/projected/3085d0d5-fb73-4f02-9dca-cdb3cf369470-kube-api-access-bkn7r\") pod \"kuadrant-operator-controller-manager-68cf589589-l2cfn\" (UID: \"3085d0d5-fb73-4f02-9dca-cdb3cf369470\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-l2cfn" Aug 05 16:44:38.438591 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:44:38.438553 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3085d0d5-fb73-4f02-9dca-cdb3cf369470-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-l2cfn\" (UID: \"3085d0d5-fb73-4f02-9dca-cdb3cf369470\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-l2cfn" Aug 05 16:44:38.438714 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:44:38.438638 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkn7r\" (UniqueName: \"kubernetes.io/projected/3085d0d5-fb73-4f02-9dca-cdb3cf369470-kube-api-access-bkn7r\") pod \"kuadrant-operator-controller-manager-68cf589589-l2cfn\" (UID: \"3085d0d5-fb73-4f02-9dca-cdb3cf369470\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-l2cfn" Aug 05 16:44:38.439020 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:44:38.438997 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3085d0d5-fb73-4f02-9dca-cdb3cf369470-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-l2cfn\" (UID: \"3085d0d5-fb73-4f02-9dca-cdb3cf369470\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-l2cfn" Aug 05 16:44:38.481111 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:44:38.481086 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkn7r\" (UniqueName: \"kubernetes.io/projected/3085d0d5-fb73-4f02-9dca-cdb3cf369470-kube-api-access-bkn7r\") pod \"kuadrant-operator-controller-manager-68cf589589-l2cfn\" (UID: \"3085d0d5-fb73-4f02-9dca-cdb3cf369470\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-l2cfn" Aug 05 16:44:38.560200 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:44:38.560169 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-l2cfn" Aug 05 16:44:38.700508 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:44:38.700475 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-l2cfn"] Aug 05 16:44:38.703958 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:44:38.703935 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3085d0d5_fb73_4f02_9dca_cdb3cf369470.slice/crio-c8aeb8b1602c2e44993e1ce32739f4b84049be0ebd9fa98284ad1dab4aedee16 WatchSource:0}: Error finding container c8aeb8b1602c2e44993e1ce32739f4b84049be0ebd9fa98284ad1dab4aedee16: Status 404 returned error can't find the container with id c8aeb8b1602c2e44993e1ce32739f4b84049be0ebd9fa98284ad1dab4aedee16 Aug 05 16:44:38.943934 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:44:38.943839 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-l2cfn" event={"ID":"3085d0d5-fb73-4f02-9dca-cdb3cf369470","Type":"ContainerStarted","Data":"e3bb466a70540dba24be89971523b78dbcf64a444eec4af9c6965065fddf04e6"} Aug 05 16:44:38.943934 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:44:38.943896 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-l2cfn" event={"ID":"3085d0d5-fb73-4f02-9dca-cdb3cf369470","Type":"ContainerStarted","Data":"c8aeb8b1602c2e44993e1ce32739f4b84049be0ebd9fa98284ad1dab4aedee16"} Aug 05 16:44:38.944148 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:44:38.943945 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-l2cfn" Aug 05 16:44:38.974492 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:44:38.974443 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-l2cfn" podStartSLOduration=0.974428421 podStartE2EDuration="974.428421ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:44:38.974312066 +0000 UTC m=+1394.131157968" watchObservedRunningTime="2026-08-05 16:44:38.974428421 +0000 UTC m=+1394.131274321" Aug 05 16:44:49.950396 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:44:49.950345 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-l2cfn" Aug 05 16:45:00.160461 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:45:00.160428 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29765805-2fgnf"] Aug 05 16:45:00.164088 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:45:00.164067 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765805-2fgnf" Aug 05 16:45:00.167522 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:45:00.167502 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q6h9g\"" Aug 05 16:45:00.173035 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:45:00.173018 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765805-2fgnf" Aug 05 16:45:00.181578 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:45:00.181548 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29765805-2fgnf"] Aug 05 16:45:00.308310 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:45:00.308282 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29765805-2fgnf"] Aug 05 16:45:00.311019 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:45:00.310994 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a997cd_da4f_4c33_ab6a_46ceb176f0e4.slice/crio-05d265f457d92353be2f9bacdd48d8534562b0a4cfcb2e177996f9cc4ae20d82 WatchSource:0}: Error finding container 05d265f457d92353be2f9bacdd48d8534562b0a4cfcb2e177996f9cc4ae20d82: Status 404 returned error can't find the container with id 05d265f457d92353be2f9bacdd48d8534562b0a4cfcb2e177996f9cc4ae20d82 Aug 05 16:45:00.313011 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:45:00.312991 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 05 16:45:01.034516 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:45:01.034482 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765805-2fgnf" event={"ID":"d5a997cd-da4f-4c33-ab6a-46ceb176f0e4","Type":"ContainerStarted","Data":"05d265f457d92353be2f9bacdd48d8534562b0a4cfcb2e177996f9cc4ae20d82"} Aug 05 16:45:04.050920 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:45:04.050888 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765805-2fgnf" event={"ID":"d5a997cd-da4f-4c33-ab6a-46ceb176f0e4","Type":"ContainerStarted","Data":"0da2b2457da04b79bcb838a31bfbbf5f7fa374ba8d498729a4c36c4afbcc5f7d"} Aug 05 16:45:05.056670 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:45:05.056629 2573 generic.go:356] "Generic (PLEG): container finished" podID="d5a997cd-da4f-4c33-ab6a-46ceb176f0e4" containerID="0da2b2457da04b79bcb838a31bfbbf5f7fa374ba8d498729a4c36c4afbcc5f7d" exitCode=0 Aug 05 16:45:05.057056 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:45:05.056711 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765805-2fgnf" event={"ID":"d5a997cd-da4f-4c33-ab6a-46ceb176f0e4","Type":"ContainerDied","Data":"0da2b2457da04b79bcb838a31bfbbf5f7fa374ba8d498729a4c36c4afbcc5f7d"} Aug 05 16:45:05.182634 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:45:05.182609 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765805-2fgnf" Aug 05 16:45:06.062019 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:45:06.061983 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765805-2fgnf" event={"ID":"d5a997cd-da4f-4c33-ab6a-46ceb176f0e4","Type":"ContainerDied","Data":"05d265f457d92353be2f9bacdd48d8534562b0a4cfcb2e177996f9cc4ae20d82"} Aug 05 16:45:06.062019 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:45:06.062018 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765805-2fgnf" Aug 05 16:45:06.062516 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:45:06.062032 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05d265f457d92353be2f9bacdd48d8534562b0a4cfcb2e177996f9cc4ae20d82" Aug 05 16:46:12.122507 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:46:12.122472 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:47:15.885346 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:47:15.885260 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:47:33.107302 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:47:33.107262 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:47:43.893139 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:47:43.893073 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:48:15.689117 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:15.689085 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx"] Aug 05 16:48:15.689613 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:15.689395 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5a997cd-da4f-4c33-ab6a-46ceb176f0e4" containerName="cleanup" Aug 05 16:48:15.689613 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:15.689406 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5a997cd-da4f-4c33-ab6a-46ceb176f0e4" containerName="cleanup" Aug 05 16:48:15.689613 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:15.689488 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5a997cd-da4f-4c33-ab6a-46ceb176f0e4" containerName="cleanup" Aug 05 16:48:15.692007 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:15.691991 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" Aug 05 16:48:15.695948 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:15.695931 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-2313f09c-kserve-self-signed-certs\"" Aug 05 16:48:15.715463 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:15.715429 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:48:15.716442 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:15.716414 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghd8q\" (UniqueName: \"kubernetes.io/projected/420594ab-444d-4112-b850-38a45e67f9ef-kube-api-access-ghd8q\") pod \"e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx\" (UID: \"420594ab-444d-4112-b850-38a45e67f9ef\") " pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" Aug 05 16:48:15.716506 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:15.716449 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/420594ab-444d-4112-b850-38a45e67f9ef-dshm\") pod \"e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx\" (UID: \"420594ab-444d-4112-b850-38a45e67f9ef\") " pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" Aug 05 16:48:15.716506 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:15.716468 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/420594ab-444d-4112-b850-38a45e67f9ef-tmp-dir\") pod \"e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx\" (UID: \"420594ab-444d-4112-b850-38a45e67f9ef\") " pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" Aug 05 16:48:15.716506 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:15.716488 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/420594ab-444d-4112-b850-38a45e67f9ef-tls-certs\") pod \"e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx\" (UID: \"420594ab-444d-4112-b850-38a45e67f9ef\") " pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" Aug 05 16:48:15.716605 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:15.716528 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/420594ab-444d-4112-b850-38a45e67f9ef-model-cache\") pod \"e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx\" (UID: \"420594ab-444d-4112-b850-38a45e67f9ef\") " pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" Aug 05 16:48:15.716605 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:15.716591 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/420594ab-444d-4112-b850-38a45e67f9ef-home\") pod \"e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx\" (UID: \"420594ab-444d-4112-b850-38a45e67f9ef\") " pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" Aug 05 16:48:15.727978 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:15.727958 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx"] Aug 05 16:48:15.817011 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:15.816973 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/420594ab-444d-4112-b850-38a45e67f9ef-dshm\") pod \"e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx\" (UID: \"420594ab-444d-4112-b850-38a45e67f9ef\") " pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" Aug 05 16:48:15.817011 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:15.817015 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/420594ab-444d-4112-b850-38a45e67f9ef-tmp-dir\") pod \"e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx\" (UID: \"420594ab-444d-4112-b850-38a45e67f9ef\") " pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" Aug 05 16:48:15.817261 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:15.817034 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/420594ab-444d-4112-b850-38a45e67f9ef-tls-certs\") pod \"e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx\" (UID: \"420594ab-444d-4112-b850-38a45e67f9ef\") " pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" Aug 05 16:48:15.817261 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:15.817074 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/420594ab-444d-4112-b850-38a45e67f9ef-model-cache\") pod \"e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx\" (UID: \"420594ab-444d-4112-b850-38a45e67f9ef\") " pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" Aug 05 16:48:15.817261 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:15.817130 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/420594ab-444d-4112-b850-38a45e67f9ef-home\") pod \"e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx\" (UID: \"420594ab-444d-4112-b850-38a45e67f9ef\") " pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" Aug 05 16:48:15.817261 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:15.817159 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghd8q\" (UniqueName: \"kubernetes.io/projected/420594ab-444d-4112-b850-38a45e67f9ef-kube-api-access-ghd8q\") pod \"e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx\" (UID: \"420594ab-444d-4112-b850-38a45e67f9ef\") " pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" Aug 05 16:48:15.817575 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:15.817546 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/420594ab-444d-4112-b850-38a45e67f9ef-tmp-dir\") pod \"e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx\" (UID: \"420594ab-444d-4112-b850-38a45e67f9ef\") " pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" Aug 05 16:48:15.817699 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:15.817657 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/420594ab-444d-4112-b850-38a45e67f9ef-home\") pod \"e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx\" (UID: \"420594ab-444d-4112-b850-38a45e67f9ef\") " pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" Aug 05 16:48:15.817759 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:15.817704 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/420594ab-444d-4112-b850-38a45e67f9ef-model-cache\") pod \"e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx\" (UID: \"420594ab-444d-4112-b850-38a45e67f9ef\") " pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" Aug 05 16:48:15.819435 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:15.819416 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/420594ab-444d-4112-b850-38a45e67f9ef-dshm\") pod \"e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx\" (UID: \"420594ab-444d-4112-b850-38a45e67f9ef\") " pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" Aug 05 16:48:15.819691 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:15.819675 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/420594ab-444d-4112-b850-38a45e67f9ef-tls-certs\") pod \"e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx\" (UID: \"420594ab-444d-4112-b850-38a45e67f9ef\") " pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" Aug 05 16:48:15.833844 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:15.833822 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghd8q\" (UniqueName: \"kubernetes.io/projected/420594ab-444d-4112-b850-38a45e67f9ef-kube-api-access-ghd8q\") pod \"e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx\" (UID: \"420594ab-444d-4112-b850-38a45e67f9ef\") " pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" Aug 05 16:48:16.000931 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:16.000844 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" Aug 05 16:48:16.137678 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:16.137655 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx"] Aug 05 16:48:16.140303 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:48:16.140281 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod420594ab_444d_4112_b850_38a45e67f9ef.slice/crio-68827133b4da03c27927168251bbd4ef81602219bff5db51928d97cab911cfca WatchSource:0}: Error finding container 68827133b4da03c27927168251bbd4ef81602219bff5db51928d97cab911cfca: Status 404 returned error can't find the container with id 68827133b4da03c27927168251bbd4ef81602219bff5db51928d97cab911cfca Aug 05 16:48:16.899580 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:16.899493 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" event={"ID":"420594ab-444d-4112-b850-38a45e67f9ef","Type":"ContainerStarted","Data":"74ab33d993f545b8e6004157494745cb5d7602976d790c483b070a529c1046ce"} Aug 05 16:48:16.899580 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:16.899533 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" event={"ID":"420594ab-444d-4112-b850-38a45e67f9ef","Type":"ContainerStarted","Data":"68827133b4da03c27927168251bbd4ef81602219bff5db51928d97cab911cfca"} Aug 05 16:48:16.922144 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:16.922093 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" podStartSLOduration=1.922078738 podStartE2EDuration="1.922078738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:48:16.920322266 +0000 UTC m=+1612.077168166" watchObservedRunningTime="2026-08-05 16:48:16.922078738 +0000 UTC m=+1612.078924638" Aug 05 16:48:26.001351 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:26.001256 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" Aug 05 16:48:26.001351 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:26.001305 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" Aug 05 16:48:26.013832 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:26.013803 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" Aug 05 16:48:26.953943 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:26.953915 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" Aug 05 16:48:29.826984 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:29.826949 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s"] Aug 05 16:48:29.830083 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:29.830067 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" Aug 05 16:48:29.835311 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:29.835277 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-2313f09c-kserve-self-signed-certs\"" Aug 05 16:48:29.848725 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:29.848702 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s"] Aug 05 16:48:29.885151 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:29.885121 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:48:29.939364 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:29.939324 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-tmp-dir\") pod \"e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s\" (UID: \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\") " pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" Aug 05 16:48:29.939564 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:29.939401 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-home\") pod \"e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s\" (UID: \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\") " pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" Aug 05 16:48:29.939564 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:29.939426 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd975\" (UniqueName: \"kubernetes.io/projected/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-kube-api-access-bd975\") pod \"e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s\" (UID: \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\") " pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" Aug 05 16:48:29.939564 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:29.939462 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-model-cache\") pod \"e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s\" (UID: \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\") " pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" Aug 05 16:48:29.939564 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:29.939548 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-tls-certs\") pod \"e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s\" (UID: \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\") " pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" Aug 05 16:48:29.939776 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:29.939590 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-dshm\") pod \"e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s\" (UID: \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\") " pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" Aug 05 16:48:29.984708 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:29.984678 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:48:30.040571 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:30.040538 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-tmp-dir\") pod \"e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s\" (UID: \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\") " pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" Aug 05 16:48:30.040750 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:30.040592 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-home\") pod \"e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s\" (UID: \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\") " pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" Aug 05 16:48:30.040750 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:30.040617 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd975\" (UniqueName: \"kubernetes.io/projected/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-kube-api-access-bd975\") pod \"e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s\" (UID: \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\") " pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" Aug 05 16:48:30.040750 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:30.040658 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-model-cache\") pod \"e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s\" (UID: \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\") " pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" Aug 05 16:48:30.040750 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:30.040714 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-tls-certs\") pod \"e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s\" (UID: \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\") " pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" Aug 05 16:48:30.040750 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:30.040746 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-dshm\") pod \"e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s\" (UID: \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\") " pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" Aug 05 16:48:30.041032 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:30.040998 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-home\") pod \"e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s\" (UID: \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\") " pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" Aug 05 16:48:30.041259 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:30.041234 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-model-cache\") pod \"e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s\" (UID: \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\") " pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" Aug 05 16:48:30.041363 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:30.041312 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-tmp-dir\") pod \"e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s\" (UID: \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\") " pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" Aug 05 16:48:30.043231 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:30.043204 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-dshm\") pod \"e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s\" (UID: \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\") " pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" Aug 05 16:48:30.043546 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:30.043526 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-tls-certs\") pod \"e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s\" (UID: \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\") " pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" Aug 05 16:48:30.050442 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:30.050419 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd975\" (UniqueName: \"kubernetes.io/projected/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-kube-api-access-bd975\") pod \"e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s\" (UID: \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\") " pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" Aug 05 16:48:30.138796 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:30.138696 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" Aug 05 16:48:30.273159 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:30.273135 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s"] Aug 05 16:48:30.275767 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:48:30.275727 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb121dfdb_ea43_4906_b9dd_7ae2aea8d1d2.slice/crio-534612674d889998223c427473352617893ce366bf3a25e7c6c8fc49673694e6 WatchSource:0}: Error finding container 534612674d889998223c427473352617893ce366bf3a25e7c6c8fc49673694e6: Status 404 returned error can't find the container with id 534612674d889998223c427473352617893ce366bf3a25e7c6c8fc49673694e6 Aug 05 16:48:30.962030 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:30.961993 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" event={"ID":"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2","Type":"ContainerStarted","Data":"9461753fbb215a2e65c4e140d5f2d569a28732a6dd4af3908f4e4f555b731cb0"} Aug 05 16:48:30.962030 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:30.962040 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" event={"ID":"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2","Type":"ContainerStarted","Data":"534612674d889998223c427473352617893ce366bf3a25e7c6c8fc49673694e6"} Aug 05 16:48:30.986432 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:30.986324 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" podStartSLOduration=1.9863092629999999 podStartE2EDuration="1.986309263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:48:30.984815967 +0000 UTC m=+1626.141661868" watchObservedRunningTime="2026-08-05 16:48:30.986309263 +0000 UTC m=+1626.143155164" Aug 05 16:48:40.139221 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:40.139179 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" Aug 05 16:48:40.139652 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:40.139233 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" Aug 05 16:48:40.151652 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:40.151627 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" Aug 05 16:48:41.016786 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:41.016759 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" Aug 05 16:48:54.080874 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:54.080834 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx"] Aug 05 16:48:54.098133 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:54.098097 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:48:55.291435 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:55.291402 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s"] Aug 05 16:48:55.304635 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:48:55.304612 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:49:09.100314 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:09.100251 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" podUID="420594ab-444d-4112-b850-38a45e67f9ef" containerName="main" containerID="cri-o://74ab33d993f545b8e6004157494745cb5d7602976d790c483b070a529c1046ce" gracePeriod=45 Aug 05 16:49:09.351847 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:09.351786 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" Aug 05 16:49:09.389635 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:09.389608 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghd8q\" (UniqueName: \"kubernetes.io/projected/420594ab-444d-4112-b850-38a45e67f9ef-kube-api-access-ghd8q\") pod \"420594ab-444d-4112-b850-38a45e67f9ef\" (UID: \"420594ab-444d-4112-b850-38a45e67f9ef\") " Aug 05 16:49:09.389806 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:09.389644 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/420594ab-444d-4112-b850-38a45e67f9ef-tls-certs\") pod \"420594ab-444d-4112-b850-38a45e67f9ef\" (UID: \"420594ab-444d-4112-b850-38a45e67f9ef\") " Aug 05 16:49:09.389806 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:09.389667 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/420594ab-444d-4112-b850-38a45e67f9ef-dshm\") pod \"420594ab-444d-4112-b850-38a45e67f9ef\" (UID: \"420594ab-444d-4112-b850-38a45e67f9ef\") " Aug 05 16:49:09.389806 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:09.389694 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/420594ab-444d-4112-b850-38a45e67f9ef-model-cache\") pod \"420594ab-444d-4112-b850-38a45e67f9ef\" (UID: \"420594ab-444d-4112-b850-38a45e67f9ef\") " Aug 05 16:49:09.389806 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:09.389722 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/420594ab-444d-4112-b850-38a45e67f9ef-tmp-dir\") pod \"420594ab-444d-4112-b850-38a45e67f9ef\" (UID: \"420594ab-444d-4112-b850-38a45e67f9ef\") " Aug 05 16:49:09.389806 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:09.389745 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/420594ab-444d-4112-b850-38a45e67f9ef-home\") pod \"420594ab-444d-4112-b850-38a45e67f9ef\" (UID: \"420594ab-444d-4112-b850-38a45e67f9ef\") " Aug 05 16:49:09.390067 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:09.389932 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420594ab-444d-4112-b850-38a45e67f9ef-model-cache" (OuterVolumeSpecName: "model-cache") pod "420594ab-444d-4112-b850-38a45e67f9ef" (UID: "420594ab-444d-4112-b850-38a45e67f9ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:49:09.390067 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:09.389972 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420594ab-444d-4112-b850-38a45e67f9ef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "420594ab-444d-4112-b850-38a45e67f9ef" (UID: "420594ab-444d-4112-b850-38a45e67f9ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:49:09.390067 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:09.390023 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/420594ab-444d-4112-b850-38a45e67f9ef-model-cache\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:49:09.390067 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:09.390038 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/420594ab-444d-4112-b850-38a45e67f9ef-tmp-dir\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:49:09.390232 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:09.390113 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420594ab-444d-4112-b850-38a45e67f9ef-home" (OuterVolumeSpecName: "home") pod "420594ab-444d-4112-b850-38a45e67f9ef" (UID: "420594ab-444d-4112-b850-38a45e67f9ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:49:09.391933 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:09.391907 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/420594ab-444d-4112-b850-38a45e67f9ef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "420594ab-444d-4112-b850-38a45e67f9ef" (UID: "420594ab-444d-4112-b850-38a45e67f9ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 16:49:09.391933 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:09.391916 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420594ab-444d-4112-b850-38a45e67f9ef-dshm" (OuterVolumeSpecName: "dshm") pod "420594ab-444d-4112-b850-38a45e67f9ef" (UID: "420594ab-444d-4112-b850-38a45e67f9ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:49:09.391933 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:09.391926 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/420594ab-444d-4112-b850-38a45e67f9ef-kube-api-access-ghd8q" (OuterVolumeSpecName: "kube-api-access-ghd8q") pod "420594ab-444d-4112-b850-38a45e67f9ef" (UID: "420594ab-444d-4112-b850-38a45e67f9ef"). InnerVolumeSpecName "kube-api-access-ghd8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:49:09.491207 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:09.491159 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/420594ab-444d-4112-b850-38a45e67f9ef-home\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:49:09.491207 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:09.491201 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghd8q\" (UniqueName: \"kubernetes.io/projected/420594ab-444d-4112-b850-38a45e67f9ef-kube-api-access-ghd8q\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:49:09.491207 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:09.491214 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/420594ab-444d-4112-b850-38a45e67f9ef-tls-certs\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:49:09.491207 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:09.491224 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/420594ab-444d-4112-b850-38a45e67f9ef-dshm\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:49:10.138721 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.138687 2573 generic.go:356] "Generic (PLEG): container finished" podID="420594ab-444d-4112-b850-38a45e67f9ef" containerID="74ab33d993f545b8e6004157494745cb5d7602976d790c483b070a529c1046ce" exitCode=0 Aug 05 16:49:10.139162 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.138753 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" Aug 05 16:49:10.139162 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.138770 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" event={"ID":"420594ab-444d-4112-b850-38a45e67f9ef","Type":"ContainerDied","Data":"74ab33d993f545b8e6004157494745cb5d7602976d790c483b070a529c1046ce"} Aug 05 16:49:10.139162 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.138810 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx" event={"ID":"420594ab-444d-4112-b850-38a45e67f9ef","Type":"ContainerDied","Data":"68827133b4da03c27927168251bbd4ef81602219bff5db51928d97cab911cfca"} Aug 05 16:49:10.139162 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.138829 2573 scope.go:117] "RemoveContainer" containerID="74ab33d993f545b8e6004157494745cb5d7602976d790c483b070a529c1046ce" Aug 05 16:49:10.147750 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.147736 2573 scope.go:117] "RemoveContainer" containerID="74ab33d993f545b8e6004157494745cb5d7602976d790c483b070a529c1046ce" Aug 05 16:49:10.148005 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:49:10.147981 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74ab33d993f545b8e6004157494745cb5d7602976d790c483b070a529c1046ce\": container with ID starting with 74ab33d993f545b8e6004157494745cb5d7602976d790c483b070a529c1046ce not found: ID does not exist" containerID="74ab33d993f545b8e6004157494745cb5d7602976d790c483b070a529c1046ce" Aug 05 16:49:10.148047 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.148011 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74ab33d993f545b8e6004157494745cb5d7602976d790c483b070a529c1046ce"} err="failed to get container status \"74ab33d993f545b8e6004157494745cb5d7602976d790c483b070a529c1046ce\": rpc error: code = NotFound desc = could not find container \"74ab33d993f545b8e6004157494745cb5d7602976d790c483b070a529c1046ce\": container with ID starting with 74ab33d993f545b8e6004157494745cb5d7602976d790c483b070a529c1046ce not found: ID does not exist" Aug 05 16:49:10.163242 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.163218 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx"] Aug 05 16:49:10.167885 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.167862 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-2313f09c-kserve-dcc647979-tvnqx"] Aug 05 16:49:10.309927 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.309884 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" podUID="b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2" containerName="main" containerID="cri-o://9461753fbb215a2e65c4e140d5f2d569a28732a6dd4af3908f4e4f555b731cb0" gracePeriod=45 Aug 05 16:49:10.556890 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.556867 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" Aug 05 16:49:10.601545 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.601508 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-model-cache\") pod \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\" (UID: \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\") " Aug 05 16:49:10.601545 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.601553 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-tls-certs\") pod \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\" (UID: \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\") " Aug 05 16:49:10.601779 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.601572 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-dshm\") pod \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\" (UID: \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\") " Aug 05 16:49:10.601779 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.601611 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-home\") pod \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\" (UID: \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\") " Aug 05 16:49:10.601779 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.601649 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-tmp-dir\") pod \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\" (UID: \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\") " Aug 05 16:49:10.601779 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.601711 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bd975\" (UniqueName: \"kubernetes.io/projected/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-kube-api-access-bd975\") pod \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\" (UID: \"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2\") " Aug 05 16:49:10.601990 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.601798 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-model-cache" (OuterVolumeSpecName: "model-cache") pod "b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2" (UID: "b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:49:10.601990 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.601861 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-home" (OuterVolumeSpecName: "home") pod "b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2" (UID: "b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:49:10.602095 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.601992 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-model-cache\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:49:10.602095 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.602009 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-home\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:49:10.602095 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.602010 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2" (UID: "b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:49:10.603720 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.603699 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-dshm" (OuterVolumeSpecName: "dshm") pod "b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2" (UID: "b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:49:10.603841 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.603730 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2" (UID: "b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 16:49:10.603841 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.603822 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-kube-api-access-bd975" (OuterVolumeSpecName: "kube-api-access-bd975") pod "b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2" (UID: "b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2"). InnerVolumeSpecName "kube-api-access-bd975". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:49:10.702703 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.702613 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd975\" (UniqueName: \"kubernetes.io/projected/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-kube-api-access-bd975\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:49:10.702703 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.702644 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-tls-certs\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:49:10.702703 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.702654 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-dshm\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:49:10.702703 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:10.702662 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2-tmp-dir\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:49:11.144939 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:11.144904 2573 generic.go:356] "Generic (PLEG): container finished" podID="b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2" containerID="9461753fbb215a2e65c4e140d5f2d569a28732a6dd4af3908f4e4f555b731cb0" exitCode=0 Aug 05 16:49:11.145438 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:11.144953 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" Aug 05 16:49:11.145438 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:11.144983 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" event={"ID":"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2","Type":"ContainerDied","Data":"9461753fbb215a2e65c4e140d5f2d569a28732a6dd4af3908f4e4f555b731cb0"} Aug 05 16:49:11.145438 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:11.145028 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s" event={"ID":"b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2","Type":"ContainerDied","Data":"534612674d889998223c427473352617893ce366bf3a25e7c6c8fc49673694e6"} Aug 05 16:49:11.145438 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:11.145046 2573 scope.go:117] "RemoveContainer" containerID="9461753fbb215a2e65c4e140d5f2d569a28732a6dd4af3908f4e4f555b731cb0" Aug 05 16:49:11.153458 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:11.153439 2573 scope.go:117] "RemoveContainer" containerID="9461753fbb215a2e65c4e140d5f2d569a28732a6dd4af3908f4e4f555b731cb0" Aug 05 16:49:11.153713 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:49:11.153692 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9461753fbb215a2e65c4e140d5f2d569a28732a6dd4af3908f4e4f555b731cb0\": container with ID starting with 9461753fbb215a2e65c4e140d5f2d569a28732a6dd4af3908f4e4f555b731cb0 not found: ID does not exist" containerID="9461753fbb215a2e65c4e140d5f2d569a28732a6dd4af3908f4e4f555b731cb0" Aug 05 16:49:11.153770 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:11.153723 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9461753fbb215a2e65c4e140d5f2d569a28732a6dd4af3908f4e4f555b731cb0"} err="failed to get container status \"9461753fbb215a2e65c4e140d5f2d569a28732a6dd4af3908f4e4f555b731cb0\": rpc error: code = NotFound desc = could not find container \"9461753fbb215a2e65c4e140d5f2d569a28732a6dd4af3908f4e4f555b731cb0\": container with ID starting with 9461753fbb215a2e65c4e140d5f2d569a28732a6dd4af3908f4e4f555b731cb0 not found: ID does not exist" Aug 05 16:49:11.173045 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:11.173014 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s"] Aug 05 16:49:11.177999 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:11.177976 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-2313f09c-kserve-dc59bc4c9-2br5s"] Aug 05 16:49:11.303612 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:11.303583 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="420594ab-444d-4112-b850-38a45e67f9ef" path="/var/lib/kubelet/pods/420594ab-444d-4112-b850-38a45e67f9ef/volumes" Aug 05 16:49:11.303981 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:49:11.303968 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2" path="/var/lib/kubelet/pods/b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2/volumes" Aug 05 16:51:15.977257 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:51:15.977203 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:52:35.640983 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:52:35.640945 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:01.405289 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:01.405204 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:04.469655 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:04.469622 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:04.583963 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:04.583930 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:04.991059 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:04.991019 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:11.620747 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:11.620709 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:11.705657 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:11.705624 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:11.842156 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:11.842112 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:12.153596 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.153563 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x"] Aug 05 16:53:12.153870 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.153858 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2" containerName="main" Aug 05 16:53:12.153931 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.153872 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2" containerName="main" Aug 05 16:53:12.153931 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.153895 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="420594ab-444d-4112-b850-38a45e67f9ef" containerName="main" Aug 05 16:53:12.153931 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.153901 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="420594ab-444d-4112-b850-38a45e67f9ef" containerName="main" Aug 05 16:53:12.154132 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.153948 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="420594ab-444d-4112-b850-38a45e67f9ef" containerName="main" Aug 05 16:53:12.154132 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.153954 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b121dfdb-ea43-4906-b9dd-7ae2aea8d1d2" containerName="main" Aug 05 16:53:12.156318 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.156302 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.159624 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.159601 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ffdafb25-openshift-default-dockercfg-98g49\"" Aug 05 16:53:12.180842 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.180794 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:12.190084 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.190060 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x"] Aug 05 16:53:12.288710 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.288670 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-credential-socket\") pod \"e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.288882 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.288720 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istio-podinfo\") pod \"e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.288882 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.288750 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt5mx\" (UniqueName: \"kubernetes.io/projected/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-kube-api-access-dt5mx\") pod \"e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.288882 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.288790 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-workload-certs\") pod \"e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.288882 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.288846 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istio-envoy\") pod \"e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.289032 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.288901 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istiod-ca-cert\") pod \"e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.289032 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.288940 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-workload-socket\") pod \"e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.289032 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.288964 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istio-data\") pod \"e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.289032 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.289009 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istio-token\") pod \"e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.389725 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.389686 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istiod-ca-cert\") pod \"e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.389920 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.389746 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-workload-socket\") pod \"e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.389920 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.389778 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istio-data\") pod \"e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.389920 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.389815 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istio-token\") pod \"e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.389920 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.389857 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-credential-socket\") pod \"e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.389920 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.389885 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istio-podinfo\") pod \"e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.389920 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.389909 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt5mx\" (UniqueName: \"kubernetes.io/projected/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-kube-api-access-dt5mx\") pod \"e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.390233 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.389951 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-workload-certs\") pod \"e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.390233 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.389977 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istio-envoy\") pod \"e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.390233 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.390164 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-workload-socket\") pod \"e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.390233 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.390216 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istio-data\") pod \"e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.390462 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.390256 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-credential-socket\") pod \"e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.390519 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.390498 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istiod-ca-cert\") pod \"e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.390566 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.390551 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-workload-certs\") pod \"e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.392223 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.392201 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istio-envoy\") pod \"e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.392503 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.392483 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istio-podinfo\") pod \"e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.400848 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.400814 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istio-token\") pod \"e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.401044 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.401027 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt5mx\" (UniqueName: \"kubernetes.io/projected/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-kube-api-access-dt5mx\") pod \"e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.466743 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.466664 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:12.615598 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.615569 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x"] Aug 05 16:53:12.616793 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:53:12.616765 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c33d62_c664_469b_aaa8_75d3a2cc1ca4.slice/crio-434ee702235dca3ae86bd93ab06fed764495aae3d473451e0da2772fa2d67df2 WatchSource:0}: Error finding container 434ee702235dca3ae86bd93ab06fed764495aae3d473451e0da2772fa2d67df2: Status 404 returned error can't find the container with id 434ee702235dca3ae86bd93ab06fed764495aae3d473451e0da2772fa2d67df2 Aug 05 16:53:12.619010 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.618976 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:53:12.619082 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.619051 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:53:12.619120 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.619090 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:53:12.923112 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:12.923078 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:13.198910 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:13.198809 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" event={"ID":"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4","Type":"ContainerStarted","Data":"0774c0be8e7d32874c344f0aad156115265808343607e03195ab7e7f582f0146"} Aug 05 16:53:13.198910 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:13.198880 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" event={"ID":"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4","Type":"ContainerStarted","Data":"434ee702235dca3ae86bd93ab06fed764495aae3d473451e0da2772fa2d67df2"} Aug 05 16:53:13.467496 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:13.467412 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:13.472245 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:13.472217 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:13.515485 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:13.515437 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" podStartSLOduration=1.51542211 podStartE2EDuration="1.51542211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:53:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:53:13.235517012 +0000 UTC m=+1908.392362914" watchObservedRunningTime="2026-08-05 16:53:13.51542211 +0000 UTC m=+1908.672268011" Aug 05 16:53:14.203207 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:14.203181 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:14.204069 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:14.204050 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:14.775909 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:14.775874 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:14.931179 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:14.931146 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:16.091990 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:16.091947 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:16.356405 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:16.356306 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:16.874629 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:16.874588 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:16.978780 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:16.978745 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:29.204534 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:29.204451 2573 patch_prober.go:28] interesting pod/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Aug 05 16:53:29.204930 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:29.204532 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" podUID="d5c33d62-c664-469b-aaa8-75d3a2cc1ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Aug 05 16:53:29.284917 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:29.284883 2573 generic.go:356] "Generic (PLEG): container finished" podID="d5c33d62-c664-469b-aaa8-75d3a2cc1ca4" containerID="0774c0be8e7d32874c344f0aad156115265808343607e03195ab7e7f582f0146" exitCode=0 Aug 05 16:53:29.285097 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:29.284917 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" event={"ID":"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4","Type":"ContainerDied","Data":"0774c0be8e7d32874c344f0aad156115265808343607e03195ab7e7f582f0146"} Aug 05 16:53:29.285331 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:29.285316 2573 scope.go:117] "RemoveContainer" containerID="0774c0be8e7d32874c344f0aad156115265808343607e03195ab7e7f582f0146" Aug 05 16:53:29.286295 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:29.286268 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:53:29.286404 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:29.286349 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:53:29.286471 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:29.286413 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:53:30.290935 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:30.290905 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" event={"ID":"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4","Type":"ContainerStarted","Data":"6a3e401c0d2badfd2ab5c455bb440358914b29358cfeb3822ae7b8e4a8033e3e"} Aug 05 16:53:30.467148 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:30.467107 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:30.467401 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:30.467356 2573 patch_prober.go:28] interesting pod/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Aug 05 16:53:30.467495 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:30.467438 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" podUID="d5c33d62-c664-469b-aaa8-75d3a2cc1ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Aug 05 16:53:31.467391 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:31.467345 2573 patch_prober.go:28] interesting pod/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Aug 05 16:53:31.467812 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:31.467417 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" podUID="d5c33d62-c664-469b-aaa8-75d3a2cc1ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Aug 05 16:53:32.467189 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:32.467150 2573 patch_prober.go:28] interesting pod/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Aug 05 16:53:32.467401 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:32.467214 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" podUID="d5c33d62-c664-469b-aaa8-75d3a2cc1ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Aug 05 16:53:33.467445 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:33.467402 2573 patch_prober.go:28] interesting pod/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Aug 05 16:53:33.467837 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:33.467473 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" podUID="d5c33d62-c664-469b-aaa8-75d3a2cc1ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Aug 05 16:53:34.467605 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:34.467571 2573 patch_prober.go:28] interesting pod/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Aug 05 16:53:34.467962 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:34.467628 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" podUID="d5c33d62-c664-469b-aaa8-75d3a2cc1ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Aug 05 16:53:35.471921 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:35.471892 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:35.472270 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:35.472146 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:35.472808 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:35.472791 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:40.144670 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:40.144639 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffdafb25-6c9z9"] Aug 05 16:53:40.148017 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:40.148000 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffdafb25-6c9z9" Aug 05 16:53:40.152613 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:40.152595 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q6h9g\"" Aug 05 16:53:40.152721 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:40.152640 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 05 16:53:40.159822 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:40.159798 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffdafb25-6c9z9"] Aug 05 16:53:40.236921 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:40.236888 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/671f3930-a1fd-49a6-917b-a3fc209e3c88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ffdafb25-6c9z9\" (UID: \"671f3930-a1fd-49a6-917b-a3fc209e3c88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffdafb25-6c9z9" Aug 05 16:53:40.338268 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:40.338231 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/671f3930-a1fd-49a6-917b-a3fc209e3c88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ffdafb25-6c9z9\" (UID: \"671f3930-a1fd-49a6-917b-a3fc209e3c88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffdafb25-6c9z9" Aug 05 16:53:40.338889 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:40.338865 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/671f3930-a1fd-49a6-917b-a3fc209e3c88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ffdafb25-6c9z9\" (UID: \"671f3930-a1fd-49a6-917b-a3fc209e3c88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffdafb25-6c9z9" Aug 05 16:53:40.458938 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:40.458852 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffdafb25-6c9z9" Aug 05 16:53:40.794169 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:40.794145 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffdafb25-6c9z9"] Aug 05 16:53:40.796144 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:53:40.796118 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod671f3930_a1fd_49a6_917b_a3fc209e3c88.slice/crio-cd684b4d7aa8468b959d0343fd08e8cf78d3b492c1d1da1db755ab4052ec9411 WatchSource:0}: Error finding container cd684b4d7aa8468b959d0343fd08e8cf78d3b492c1d1da1db755ab4052ec9411: Status 404 returned error can't find the container with id cd684b4d7aa8468b959d0343fd08e8cf78d3b492c1d1da1db755ab4052ec9411 Aug 05 16:53:41.338483 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:41.338389 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffdafb25-6c9z9" event={"ID":"671f3930-a1fd-49a6-917b-a3fc209e3c88","Type":"ContainerStarted","Data":"17e6ffda7bd4e030b589743f692738ec5bda10d4c8e5f270d3ebda4ff8903e51"} Aug 05 16:53:41.338483 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:41.338457 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffdafb25-6c9z9" event={"ID":"671f3930-a1fd-49a6-917b-a3fc209e3c88","Type":"ContainerStarted","Data":"cd684b4d7aa8468b959d0343fd08e8cf78d3b492c1d1da1db755ab4052ec9411"} Aug 05 16:53:41.366225 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:41.366177 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffdafb25-6c9z9" podStartSLOduration=1.366162901 podStartE2EDuration="1.366162901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:53:41.365549979 +0000 UTC m=+1936.522395879" watchObservedRunningTime="2026-08-05 16:53:41.366162901 +0000 UTC m=+1936.523008795" Aug 05 16:53:42.344607 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:42.344570 2573 generic.go:356] "Generic (PLEG): container finished" podID="671f3930-a1fd-49a6-917b-a3fc209e3c88" containerID="17e6ffda7bd4e030b589743f692738ec5bda10d4c8e5f270d3ebda4ff8903e51" exitCode=0 Aug 05 16:53:42.345013 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:42.344651 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffdafb25-6c9z9" event={"ID":"671f3930-a1fd-49a6-917b-a3fc209e3c88","Type":"ContainerDied","Data":"17e6ffda7bd4e030b589743f692738ec5bda10d4c8e5f270d3ebda4ff8903e51"} Aug 05 16:53:43.483091 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:43.483066 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffdafb25-6c9z9" Aug 05 16:53:43.568349 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:43.568316 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/671f3930-a1fd-49a6-917b-a3fc209e3c88-maas-api-service-ca\") pod \"671f3930-a1fd-49a6-917b-a3fc209e3c88\" (UID: \"671f3930-a1fd-49a6-917b-a3fc209e3c88\") " Aug 05 16:53:43.568716 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:43.568694 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/671f3930-a1fd-49a6-917b-a3fc209e3c88-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "671f3930-a1fd-49a6-917b-a3fc209e3c88" (UID: "671f3930-a1fd-49a6-917b-a3fc209e3c88"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:53:43.669797 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:43.669717 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/671f3930-a1fd-49a6-917b-a3fc209e3c88-maas-api-service-ca\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:53:44.354702 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:44.354668 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffdafb25-6c9z9" Aug 05 16:53:44.354880 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:44.354674 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffdafb25-6c9z9" event={"ID":"671f3930-a1fd-49a6-917b-a3fc209e3c88","Type":"ContainerDied","Data":"cd684b4d7aa8468b959d0343fd08e8cf78d3b492c1d1da1db755ab4052ec9411"} Aug 05 16:53:44.354880 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:44.354780 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd684b4d7aa8468b959d0343fd08e8cf78d3b492c1d1da1db755ab4052ec9411" Aug 05 16:53:46.822829 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:46.822796 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:46.920387 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:46.920351 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:47.025700 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:47.025666 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:47.112205 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:47.112108 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffdafb25-6c9z9"] Aug 05 16:53:47.115436 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:47.115406 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffdafb25-6c9z9"] Aug 05 16:53:47.179766 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:47.179675 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:47.288974 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:47.288940 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x"] Aug 05 16:53:47.289171 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:47.289148 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" podUID="d5c33d62-c664-469b-aaa8-75d3a2cc1ca4" containerName="istio-proxy" containerID="cri-o://6a3e401c0d2badfd2ab5c455bb440358914b29358cfeb3822ae7b8e4a8033e3e" gracePeriod=30 Aug 05 16:53:47.304105 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:47.304079 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="671f3930-a1fd-49a6-917b-a3fc209e3c88" path="/var/lib/kubelet/pods/671f3930-a1fd-49a6-917b-a3fc209e3c88/volumes" Aug 05 16:53:47.329576 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:47.329543 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:47.555741 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:47.555695 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:47.796436 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:47.796402 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:47.965804 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:47.965716 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:48.096218 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.096182 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:48.381089 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.381060 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k"] Aug 05 16:53:48.381401 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.381388 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="671f3930-a1fd-49a6-917b-a3fc209e3c88" containerName="revoke-keys" Aug 05 16:53:48.381459 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.381403 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="671f3930-a1fd-49a6-917b-a3fc209e3c88" containerName="revoke-keys" Aug 05 16:53:48.381459 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.381452 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="671f3930-a1fd-49a6-917b-a3fc209e3c88" containerName="revoke-keys" Aug 05 16:53:48.384395 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.384366 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.390273 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.390239 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:48.406847 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.406823 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-bb276d9d-gw-openshift-default-dockercfg-bkvd6\"" Aug 05 16:53:48.409116 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.409091 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istio-envoy\") pod \"e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.409219 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.409130 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istio-data\") pod \"e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.409219 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.409168 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istio-token\") pod \"e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.409219 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.409197 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istio-podinfo\") pod \"e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.409355 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.409240 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm9kp\" (UniqueName: \"kubernetes.io/projected/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-kube-api-access-vm9kp\") pod \"e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.409355 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.409281 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istiod-ca-cert\") pod \"e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.409481 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.409416 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-workload-socket\") pod \"e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.409481 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.409452 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-credential-socket\") pod \"e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.409589 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.409484 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-workload-certs\") pod \"e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.450839 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.450803 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k"] Aug 05 16:53:48.510061 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.510026 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-workload-socket\") pod \"e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.510061 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.510062 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-credential-socket\") pod \"e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.510327 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.510083 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-workload-certs\") pod \"e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.510327 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.510131 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istio-envoy\") pod \"e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.510327 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.510157 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istio-data\") pod \"e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.510327 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.510205 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istio-token\") pod \"e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.510327 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.510238 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istio-podinfo\") pod \"e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.510327 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.510275 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm9kp\" (UniqueName: \"kubernetes.io/projected/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-kube-api-access-vm9kp\") pod \"e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.510327 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.510311 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istiod-ca-cert\") pod \"e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.510697 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.510523 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-workload-socket\") pod \"e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.510697 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.510583 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-workload-certs\") pod \"e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.510697 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.510602 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-credential-socket\") pod \"e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.510900 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.510876 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istio-data\") pod \"e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.511100 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.511077 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istiod-ca-cert\") pod \"e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.512633 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.512604 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istio-envoy\") pod \"e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.512878 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.512858 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istio-podinfo\") pod \"e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.525870 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.525843 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istio-token\") pod \"e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.527778 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.527748 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm9kp\" (UniqueName: \"kubernetes.io/projected/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-kube-api-access-vm9kp\") pod \"e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.647618 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.647531 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:48.695258 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.695222 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:48.835357 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.835326 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k"] Aug 05 16:53:48.836460 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:53:48.836425 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaeb09f4_5bad_4a7e_87c7_2cda2a2f2e29.slice/crio-404ea6bfae9eca14af7595c6990d00f9e4407faf38f0f97d3218432d177f3e04 WatchSource:0}: Error finding container 404ea6bfae9eca14af7595c6990d00f9e4407faf38f0f97d3218432d177f3e04: Status 404 returned error can't find the container with id 404ea6bfae9eca14af7595c6990d00f9e4407faf38f0f97d3218432d177f3e04 Aug 05 16:53:48.838759 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.838728 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:53:48.838854 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.838803 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:53:48.838929 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:48.838852 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:53:49.382090 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:49.382051 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" event={"ID":"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29","Type":"ContainerStarted","Data":"b632ac469333d635fdfb17840c3db93ecf1daa52214f765957a39a7565bdf0c2"} Aug 05 16:53:49.382492 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:49.382103 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" event={"ID":"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29","Type":"ContainerStarted","Data":"404ea6bfae9eca14af7595c6990d00f9e4407faf38f0f97d3218432d177f3e04"} Aug 05 16:53:49.415978 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:49.415922 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" podStartSLOduration=1.415906463 podStartE2EDuration="1.415906463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:53:49.415035704 +0000 UTC m=+1944.571881615" watchObservedRunningTime="2026-08-05 16:53:49.415906463 +0000 UTC m=+1944.572752417" Aug 05 16:53:49.696038 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:49.695952 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:49.701126 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:49.701103 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:50.386856 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:50.386828 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:50.387858 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:50.387837 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:53:50.474279 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:50.474244 2573 patch_prober.go:28] interesting pod/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 05 16:53:50.474424 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:50.474307 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" podUID="d5c33d62-c664-469b-aaa8-75d3a2cc1ca4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 05 16:53:50.925162 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:50.925119 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:51.191318 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:51.191223 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:51.417459 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:51.417422 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:51.519245 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:51.519209 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:51.864637 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:51.864520 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:51.961225 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:51.961183 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:53:52.544333 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:52.544304 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:52.647144 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:52.647049 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-credential-socket\") pod \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " Aug 05 16:53:52.647144 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:52.647097 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istio-envoy\") pod \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " Aug 05 16:53:52.647144 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:52.647116 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-workload-socket\") pod \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " Aug 05 16:53:52.647477 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:52.647169 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dt5mx\" (UniqueName: \"kubernetes.io/projected/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-kube-api-access-dt5mx\") pod \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " Aug 05 16:53:52.647477 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:52.647196 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istio-data\") pod \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " Aug 05 16:53:52.647477 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:52.647223 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-workload-certs\") pod \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " Aug 05 16:53:52.647477 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:52.647249 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istio-podinfo\") pod \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " Aug 05 16:53:52.647477 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:52.647271 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istio-token\") pod \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " Aug 05 16:53:52.647477 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:52.647333 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istiod-ca-cert\") pod \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\" (UID: \"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4\") " Aug 05 16:53:52.647477 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:52.647399 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d5c33d62-c664-469b-aaa8-75d3a2cc1ca4" (UID: "d5c33d62-c664-469b-aaa8-75d3a2cc1ca4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:53:52.647903 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:52.647556 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d5c33d62-c664-469b-aaa8-75d3a2cc1ca4" (UID: "d5c33d62-c664-469b-aaa8-75d3a2cc1ca4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:53:52.647903 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:52.647683 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d5c33d62-c664-469b-aaa8-75d3a2cc1ca4" (UID: "d5c33d62-c664-469b-aaa8-75d3a2cc1ca4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:53:52.647903 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:52.647723 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-credential-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:53:52.647903 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:52.647739 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-workload-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:53:52.647903 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:52.647873 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istio-data" (OuterVolumeSpecName: "istio-data") pod "d5c33d62-c664-469b-aaa8-75d3a2cc1ca4" (UID: "d5c33d62-c664-469b-aaa8-75d3a2cc1ca4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:53:52.648171 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:52.648152 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d5c33d62-c664-469b-aaa8-75d3a2cc1ca4" (UID: "d5c33d62-c664-469b-aaa8-75d3a2cc1ca4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:53:52.649700 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:52.649657 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-kube-api-access-dt5mx" (OuterVolumeSpecName: "kube-api-access-dt5mx") pod "d5c33d62-c664-469b-aaa8-75d3a2cc1ca4" (UID: "d5c33d62-c664-469b-aaa8-75d3a2cc1ca4"). InnerVolumeSpecName "kube-api-access-dt5mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:53:52.649811 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:52.649732 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d5c33d62-c664-469b-aaa8-75d3a2cc1ca4" (UID: "d5c33d62-c664-469b-aaa8-75d3a2cc1ca4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:53:52.649811 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:52.649734 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istio-token" (OuterVolumeSpecName: "istio-token") pod "d5c33d62-c664-469b-aaa8-75d3a2cc1ca4" (UID: "d5c33d62-c664-469b-aaa8-75d3a2cc1ca4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:53:52.649811 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:52.649755 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d5c33d62-c664-469b-aaa8-75d3a2cc1ca4" (UID: "d5c33d62-c664-469b-aaa8-75d3a2cc1ca4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 16:53:52.748769 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:52.748739 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istio-envoy\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:53:52.748769 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:52.748766 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt5mx\" (UniqueName: \"kubernetes.io/projected/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-kube-api-access-dt5mx\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:53:52.748769 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:52.748777 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istio-data\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:53:52.749004 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:52.748786 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-workload-certs\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:53:52.749004 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:52.748794 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istio-podinfo\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:53:52.749004 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:52.748802 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istio-token\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:53:52.749004 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:52.748809 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4-istiod-ca-cert\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:53:53.401839 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:53.401808 2573 generic.go:356] "Generic (PLEG): container finished" podID="d5c33d62-c664-469b-aaa8-75d3a2cc1ca4" containerID="6a3e401c0d2badfd2ab5c455bb440358914b29358cfeb3822ae7b8e4a8033e3e" exitCode=0 Aug 05 16:53:53.402015 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:53.401851 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" Aug 05 16:53:53.402015 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:53.401881 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" event={"ID":"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4","Type":"ContainerDied","Data":"6a3e401c0d2badfd2ab5c455bb440358914b29358cfeb3822ae7b8e4a8033e3e"} Aug 05 16:53:53.402015 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:53.401940 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x" event={"ID":"d5c33d62-c664-469b-aaa8-75d3a2cc1ca4","Type":"ContainerDied","Data":"434ee702235dca3ae86bd93ab06fed764495aae3d473451e0da2772fa2d67df2"} Aug 05 16:53:53.402015 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:53.401970 2573 scope.go:117] "RemoveContainer" containerID="6a3e401c0d2badfd2ab5c455bb440358914b29358cfeb3822ae7b8e4a8033e3e" Aug 05 16:53:53.410479 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:53.410288 2573 scope.go:117] "RemoveContainer" containerID="0774c0be8e7d32874c344f0aad156115265808343607e03195ab7e7f582f0146" Aug 05 16:53:53.417890 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:53.417861 2573 scope.go:117] "RemoveContainer" containerID="6a3e401c0d2badfd2ab5c455bb440358914b29358cfeb3822ae7b8e4a8033e3e" Aug 05 16:53:53.418110 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:53:53.418091 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a3e401c0d2badfd2ab5c455bb440358914b29358cfeb3822ae7b8e4a8033e3e\": container with ID starting with 6a3e401c0d2badfd2ab5c455bb440358914b29358cfeb3822ae7b8e4a8033e3e not found: ID does not exist" containerID="6a3e401c0d2badfd2ab5c455bb440358914b29358cfeb3822ae7b8e4a8033e3e" Aug 05 16:53:53.418211 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:53.418117 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a3e401c0d2badfd2ab5c455bb440358914b29358cfeb3822ae7b8e4a8033e3e"} err="failed to get container status \"6a3e401c0d2badfd2ab5c455bb440358914b29358cfeb3822ae7b8e4a8033e3e\": rpc error: code = NotFound desc = could not find container \"6a3e401c0d2badfd2ab5c455bb440358914b29358cfeb3822ae7b8e4a8033e3e\": container with ID starting with 6a3e401c0d2badfd2ab5c455bb440358914b29358cfeb3822ae7b8e4a8033e3e not found: ID does not exist" Aug 05 16:53:53.418211 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:53.418133 2573 scope.go:117] "RemoveContainer" containerID="0774c0be8e7d32874c344f0aad156115265808343607e03195ab7e7f582f0146" Aug 05 16:53:53.418333 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:53:53.418319 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0774c0be8e7d32874c344f0aad156115265808343607e03195ab7e7f582f0146\": container with ID starting with 0774c0be8e7d32874c344f0aad156115265808343607e03195ab7e7f582f0146 not found: ID does not exist" containerID="0774c0be8e7d32874c344f0aad156115265808343607e03195ab7e7f582f0146" Aug 05 16:53:53.418392 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:53.418337 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0774c0be8e7d32874c344f0aad156115265808343607e03195ab7e7f582f0146"} err="failed to get container status \"0774c0be8e7d32874c344f0aad156115265808343607e03195ab7e7f582f0146\": rpc error: code = NotFound desc = could not find container \"0774c0be8e7d32874c344f0aad156115265808343607e03195ab7e7f582f0146\": container with ID starting with 0774c0be8e7d32874c344f0aad156115265808343607e03195ab7e7f582f0146 not found: ID does not exist" Aug 05 16:53:53.442166 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:53.442143 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x"] Aug 05 16:53:53.448689 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:53.448659 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ffdafb25-openshift-default-8454d4c688-7s85x"] Aug 05 16:53:55.308500 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:53:55.308469 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5c33d62-c664-469b-aaa8-75d3a2cc1ca4" path="/var/lib/kubelet/pods/d5c33d62-c664-469b-aaa8-75d3a2cc1ca4/volumes" Aug 05 16:54:04.454533 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:04.454498 2573 generic.go:356] "Generic (PLEG): container finished" podID="caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29" containerID="b632ac469333d635fdfb17840c3db93ecf1daa52214f765957a39a7565bdf0c2" exitCode=0 Aug 05 16:54:04.454947 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:04.454567 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" event={"ID":"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29","Type":"ContainerDied","Data":"b632ac469333d635fdfb17840c3db93ecf1daa52214f765957a39a7565bdf0c2"} Aug 05 16:54:04.454947 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:04.454927 2573 scope.go:117] "RemoveContainer" containerID="b632ac469333d635fdfb17840c3db93ecf1daa52214f765957a39a7565bdf0c2" Aug 05 16:54:04.456001 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:04.455971 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:54:04.456099 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:04.456056 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:54:04.456167 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:04.456103 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:54:05.047228 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:05.047172 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:05.387714 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:05.387607 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:54:05.461882 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:05.461852 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" event={"ID":"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29","Type":"ContainerStarted","Data":"bf88842d315c9dad4c55fc8cd3ef95c8c1259f440b972a0fca91f2398a2447db"} Aug 05 16:54:05.696009 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:05.695914 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:54:05.697073 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:05.697050 2573 patch_prober.go:28] interesting pod/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Aug 05 16:54:05.697133 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:05.697098 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" podUID="caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Aug 05 16:54:06.696068 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:06.696030 2573 patch_prober.go:28] interesting pod/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Aug 05 16:54:06.696497 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:06.696096 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" podUID="caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Aug 05 16:54:07.695616 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:07.695572 2573 patch_prober.go:28] interesting pod/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Aug 05 16:54:07.695801 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:07.695635 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" podUID="caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Aug 05 16:54:08.695971 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:08.695935 2573 patch_prober.go:28] interesting pod/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Aug 05 16:54:08.696361 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:08.695992 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" podUID="caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Aug 05 16:54:09.696237 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:09.696192 2573 patch_prober.go:28] interesting pod/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Aug 05 16:54:09.696667 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:09.696265 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" podUID="caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Aug 05 16:54:10.700452 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:10.700422 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:54:10.700829 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:10.700670 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:54:10.701419 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:10.701403 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:54:19.748988 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:19.748894 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb276d9d-mzpkz"] Aug 05 16:54:19.749610 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:19.749351 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c33d62-c664-469b-aaa8-75d3a2cc1ca4" containerName="istio-proxy" Aug 05 16:54:19.749610 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:19.749369 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c33d62-c664-469b-aaa8-75d3a2cc1ca4" containerName="istio-proxy" Aug 05 16:54:19.749610 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:19.749401 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c33d62-c664-469b-aaa8-75d3a2cc1ca4" containerName="istio-proxy" Aug 05 16:54:19.749610 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:19.749409 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c33d62-c664-469b-aaa8-75d3a2cc1ca4" containerName="istio-proxy" Aug 05 16:54:19.749610 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:19.749513 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5c33d62-c664-469b-aaa8-75d3a2cc1ca4" containerName="istio-proxy" Aug 05 16:54:19.751523 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:19.751502 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb276d9d-mzpkz" Aug 05 16:54:19.758099 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:19.758077 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 05 16:54:19.758522 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:19.758504 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q6h9g\"" Aug 05 16:54:19.766034 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:19.766006 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb276d9d-mzpkz"] Aug 05 16:54:19.893329 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:19.893293 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e9c76c8-1dcc-4bd6-b5a8-26542d69255a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bb276d9d-mzpkz\" (UID: \"4e9c76c8-1dcc-4bd6-b5a8-26542d69255a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb276d9d-mzpkz" Aug 05 16:54:19.993798 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:19.993762 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e9c76c8-1dcc-4bd6-b5a8-26542d69255a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bb276d9d-mzpkz\" (UID: \"4e9c76c8-1dcc-4bd6-b5a8-26542d69255a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb276d9d-mzpkz" Aug 05 16:54:19.994413 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:19.994365 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e9c76c8-1dcc-4bd6-b5a8-26542d69255a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bb276d9d-mzpkz\" (UID: \"4e9c76c8-1dcc-4bd6-b5a8-26542d69255a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb276d9d-mzpkz" Aug 05 16:54:20.061212 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:20.061178 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb276d9d-mzpkz" Aug 05 16:54:20.191809 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:20.191777 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb276d9d-mzpkz"] Aug 05 16:54:20.200069 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:54:20.200043 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9c76c8_1dcc_4bd6_b5a8_26542d69255a.slice/crio-37479ed5f62b50ff49daebec5e3ed1b466dd2dabbf7669a2a763a2081677da3d WatchSource:0}: Error finding container 37479ed5f62b50ff49daebec5e3ed1b466dd2dabbf7669a2a763a2081677da3d: Status 404 returned error can't find the container with id 37479ed5f62b50ff49daebec5e3ed1b466dd2dabbf7669a2a763a2081677da3d Aug 05 16:54:20.529642 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:20.529611 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb276d9d-mzpkz" event={"ID":"4e9c76c8-1dcc-4bd6-b5a8-26542d69255a","Type":"ContainerStarted","Data":"37479ed5f62b50ff49daebec5e3ed1b466dd2dabbf7669a2a763a2081677da3d"} Aug 05 16:54:21.535675 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:21.535631 2573 generic.go:356] "Generic (PLEG): container finished" podID="4e9c76c8-1dcc-4bd6-b5a8-26542d69255a" containerID="81f6bb923c4eb7e1d0bcd1dbfbdca65dc98487c397fde6faaa68b83c59080622" exitCode=0 Aug 05 16:54:21.536081 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:21.535716 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb276d9d-mzpkz" event={"ID":"4e9c76c8-1dcc-4bd6-b5a8-26542d69255a","Type":"ContainerDied","Data":"81f6bb923c4eb7e1d0bcd1dbfbdca65dc98487c397fde6faaa68b83c59080622"} Aug 05 16:54:22.665034 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:22.665008 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb276d9d-mzpkz" Aug 05 16:54:22.816832 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:22.816739 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e9c76c8-1dcc-4bd6-b5a8-26542d69255a-maas-api-service-ca\") pod \"4e9c76c8-1dcc-4bd6-b5a8-26542d69255a\" (UID: \"4e9c76c8-1dcc-4bd6-b5a8-26542d69255a\") " Aug 05 16:54:22.817167 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:22.817138 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e9c76c8-1dcc-4bd6-b5a8-26542d69255a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4e9c76c8-1dcc-4bd6-b5a8-26542d69255a" (UID: "4e9c76c8-1dcc-4bd6-b5a8-26542d69255a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:54:22.917704 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:22.917664 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e9c76c8-1dcc-4bd6-b5a8-26542d69255a-maas-api-service-ca\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:54:23.545488 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:23.545447 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb276d9d-mzpkz" event={"ID":"4e9c76c8-1dcc-4bd6-b5a8-26542d69255a","Type":"ContainerDied","Data":"37479ed5f62b50ff49daebec5e3ed1b466dd2dabbf7669a2a763a2081677da3d"} Aug 05 16:54:23.545654 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:23.545515 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37479ed5f62b50ff49daebec5e3ed1b466dd2dabbf7669a2a763a2081677da3d" Aug 05 16:54:23.545654 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:23.545469 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb276d9d-mzpkz" Aug 05 16:54:40.098045 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:40.098004 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:44.018730 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:44.018687 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:44.147472 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:44.147432 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:44.295111 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:44.295029 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb276d9d-mzpkz"] Aug 05 16:54:44.298699 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:44.298664 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:44.299148 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:44.299131 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb276d9d-mzpkz"] Aug 05 16:54:44.454586 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:44.454558 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k"] Aug 05 16:54:44.454825 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:44.454784 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" podUID="caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29" containerName="istio-proxy" containerID="cri-o://bf88842d315c9dad4c55fc8cd3ef95c8c1259f440b972a0fca91f2398a2447db" gracePeriod=30 Aug 05 16:54:44.540031 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:44.539998 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:44.826683 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:44.826647 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:44.978349 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:44.978317 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:45.275308 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.275272 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt"] Aug 05 16:54:45.275786 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.275735 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e9c76c8-1dcc-4bd6-b5a8-26542d69255a" containerName="revoke-keys" Aug 05 16:54:45.275786 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.275753 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e9c76c8-1dcc-4bd6-b5a8-26542d69255a" containerName="revoke-keys" Aug 05 16:54:45.275906 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.275846 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5c33d62-c664-469b-aaa8-75d3a2cc1ca4" containerName="istio-proxy" Aug 05 16:54:45.275906 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.275860 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e9c76c8-1dcc-4bd6-b5a8-26542d69255a" containerName="revoke-keys" Aug 05 16:54:45.277897 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.277876 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.280810 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.280786 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ff3f79bc-openshift-default-dockercfg-gvcxt\"" Aug 05 16:54:45.304897 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.304872 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e9c76c8-1dcc-4bd6-b5a8-26542d69255a" path="/var/lib/kubelet/pods/4e9c76c8-1dcc-4bd6-b5a8-26542d69255a/volumes" Aug 05 16:54:45.305182 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.305167 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt"] Aug 05 16:54:45.322107 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.322076 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:45.413726 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.413693 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5nmq\" (UniqueName: \"kubernetes.io/projected/a2300534-a518-4f0c-abb7-5d0e70d301dc-kube-api-access-n5nmq\") pod \"e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.413901 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.413743 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-istio-data\") pod \"e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.413901 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.413807 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-istio-envoy\") pod \"e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.413901 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.413829 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-credential-socket\") pod \"e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.414095 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.413907 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2300534-a518-4f0c-abb7-5d0e70d301dc-istio-token\") pod \"e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.414095 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.414003 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-workload-socket\") pod \"e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.414187 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.414097 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2300534-a518-4f0c-abb7-5d0e70d301dc-istiod-ca-cert\") pod \"e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.414235 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.414217 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-workload-certs\") pod \"e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.414292 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.414263 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2300534-a518-4f0c-abb7-5d0e70d301dc-istio-podinfo\") pod \"e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.515221 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.515187 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-credential-socket\") pod \"e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.515221 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.515224 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2300534-a518-4f0c-abb7-5d0e70d301dc-istio-token\") pod \"e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.515488 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.515248 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-workload-socket\") pod \"e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.515488 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.515273 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2300534-a518-4f0c-abb7-5d0e70d301dc-istiod-ca-cert\") pod \"e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.515488 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.515306 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-workload-certs\") pod \"e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.515488 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.515325 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2300534-a518-4f0c-abb7-5d0e70d301dc-istio-podinfo\") pod \"e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.515488 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.515387 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5nmq\" (UniqueName: \"kubernetes.io/projected/a2300534-a518-4f0c-abb7-5d0e70d301dc-kube-api-access-n5nmq\") pod \"e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.515488 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.515434 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-istio-data\") pod \"e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.515488 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.515478 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-istio-envoy\") pod \"e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.515766 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.515740 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-credential-socket\") pod \"e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.515823 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.515749 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-workload-certs\") pod \"e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.516011 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.515986 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-workload-socket\") pod \"e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.516183 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.515993 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-istio-data\") pod \"e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.516183 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.516116 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2300534-a518-4f0c-abb7-5d0e70d301dc-istiod-ca-cert\") pod \"e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.517973 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.517951 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2300534-a518-4f0c-abb7-5d0e70d301dc-istio-podinfo\") pod \"e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.518081 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.518011 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-istio-envoy\") pod \"e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.525552 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.525483 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2300534-a518-4f0c-abb7-5d0e70d301dc-istio-token\") pod \"e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.527999 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.527978 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5nmq\" (UniqueName: \"kubernetes.io/projected/a2300534-a518-4f0c-abb7-5d0e70d301dc-kube-api-access-n5nmq\") pod \"e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.588176 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.588137 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:45.732137 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.732110 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt"] Aug 05 16:54:45.734436 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:54:45.734370 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2300534_a518_4f0c_abb7_5d0e70d301dc.slice/crio-0583b52d073ecc79ce37e3fbb8e07007037ed617892e9d2a61fd003e575140b6 WatchSource:0}: Error finding container 0583b52d073ecc79ce37e3fbb8e07007037ed617892e9d2a61fd003e575140b6: Status 404 returned error can't find the container with id 0583b52d073ecc79ce37e3fbb8e07007037ed617892e9d2a61fd003e575140b6 Aug 05 16:54:45.737056 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.737028 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:54:45.737153 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.737087 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:54:45.737153 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:45.737114 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:54:46.350947 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:46.350916 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:46.650603 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:46.650512 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" event={"ID":"a2300534-a518-4f0c-abb7-5d0e70d301dc","Type":"ContainerStarted","Data":"ff98676de95bf2252eadfeb9cd4c719b0ec5900dc4b815cf211c5d63881f7754"} Aug 05 16:54:46.650603 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:46.650567 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" event={"ID":"a2300534-a518-4f0c-abb7-5d0e70d301dc","Type":"ContainerStarted","Data":"0583b52d073ecc79ce37e3fbb8e07007037ed617892e9d2a61fd003e575140b6"} Aug 05 16:54:46.676562 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:46.676512 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" podStartSLOduration=1.676493024 podStartE2EDuration="1.676493024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:54:46.674810454 +0000 UTC m=+2001.831656356" watchObservedRunningTime="2026-08-05 16:54:46.676493024 +0000 UTC m=+2001.833338924" Aug 05 16:54:47.589011 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:47.588970 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:47.593722 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:47.593701 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:47.655826 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:47.655795 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:47.656646 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:47.656628 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:54:48.100499 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:48.100468 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:48.190435 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:48.190397 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:49.248171 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.248137 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:49.503473 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.503397 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:49.666911 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.666878 2573 generic.go:356] "Generic (PLEG): container finished" podID="caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29" containerID="bf88842d315c9dad4c55fc8cd3ef95c8c1259f440b972a0fca91f2398a2447db" exitCode=0 Aug 05 16:54:49.667099 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.666913 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" event={"ID":"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29","Type":"ContainerDied","Data":"bf88842d315c9dad4c55fc8cd3ef95c8c1259f440b972a0fca91f2398a2447db"} Aug 05 16:54:49.667099 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.666984 2573 scope.go:117] "RemoveContainer" containerID="b632ac469333d635fdfb17840c3db93ecf1daa52214f765957a39a7565bdf0c2" Aug 05 16:54:49.713596 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.713574 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:54:49.795654 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.795626 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:49.855906 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.855873 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-workload-socket\") pod \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " Aug 05 16:54:49.856074 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.855936 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istio-data\") pod \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " Aug 05 16:54:49.856074 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.855976 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-credential-socket\") pod \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " Aug 05 16:54:49.856074 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.855998 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-workload-certs\") pod \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " Aug 05 16:54:49.856074 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.856020 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istio-envoy\") pod \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " Aug 05 16:54:49.856074 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.856043 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vm9kp\" (UniqueName: \"kubernetes.io/projected/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-kube-api-access-vm9kp\") pod \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " Aug 05 16:54:49.856343 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.856101 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istiod-ca-cert\") pod \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " Aug 05 16:54:49.856343 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.856193 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istio-podinfo\") pod \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " Aug 05 16:54:49.856343 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.856202 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istio-data" (OuterVolumeSpecName: "istio-data") pod "caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29" (UID: "caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:54:49.856343 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.856214 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29" (UID: "caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:54:49.856343 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.856219 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istio-token\") pod \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\" (UID: \"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29\") " Aug 05 16:54:49.856343 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.856306 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29" (UID: "caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:54:49.856676 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.856343 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29" (UID: "caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:54:49.856676 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.856588 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29" (UID: "caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:54:49.856676 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.856635 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-workload-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:54:49.856676 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.856657 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istio-data\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:54:49.856676 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.856670 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-credential-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:54:49.856883 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.856685 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-workload-certs\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:54:49.858344 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.858320 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istio-token" (OuterVolumeSpecName: "istio-token") pod "caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29" (UID: "caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:54:49.858596 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.858572 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29" (UID: "caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:54:49.858720 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.858701 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29" (UID: "caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 16:54:49.858778 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.858720 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-kube-api-access-vm9kp" (OuterVolumeSpecName: "kube-api-access-vm9kp") pod "caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29" (UID: "caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29"). InnerVolumeSpecName "kube-api-access-vm9kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:54:49.897799 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.897767 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:54:49.957453 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.957421 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istio-envoy\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:54:49.957453 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.957451 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm9kp\" (UniqueName: \"kubernetes.io/projected/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-kube-api-access-vm9kp\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:54:49.957654 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.957462 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istiod-ca-cert\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:54:49.957654 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.957472 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istio-podinfo\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:54:49.957654 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:49.957481 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29-istio-token\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:54:50.672634 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:50.672605 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" Aug 05 16:54:50.672634 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:50.672613 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k" event={"ID":"caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29","Type":"ContainerDied","Data":"404ea6bfae9eca14af7595c6990d00f9e4407faf38f0f97d3218432d177f3e04"} Aug 05 16:54:50.673097 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:50.672675 2573 scope.go:117] "RemoveContainer" containerID="bf88842d315c9dad4c55fc8cd3ef95c8c1259f440b972a0fca91f2398a2447db" Aug 05 16:54:50.703611 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:50.703579 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k"] Aug 05 16:54:50.706799 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:50.706771 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-bb276d9d-gw-openshift-default-5b86fc8d88-rrr9k"] Aug 05 16:54:51.303279 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:54:51.303242 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29" path="/var/lib/kubelet/pods/caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29/volumes" Aug 05 16:55:00.717831 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:00.717737 2573 generic.go:356] "Generic (PLEG): container finished" podID="a2300534-a518-4f0c-abb7-5d0e70d301dc" containerID="ff98676de95bf2252eadfeb9cd4c719b0ec5900dc4b815cf211c5d63881f7754" exitCode=0 Aug 05 16:55:00.717831 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:00.717809 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" event={"ID":"a2300534-a518-4f0c-abb7-5d0e70d301dc","Type":"ContainerDied","Data":"ff98676de95bf2252eadfeb9cd4c719b0ec5900dc4b815cf211c5d63881f7754"} Aug 05 16:55:00.718244 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:00.718170 2573 scope.go:117] "RemoveContainer" containerID="ff98676de95bf2252eadfeb9cd4c719b0ec5900dc4b815cf211c5d63881f7754" Aug 05 16:55:00.719224 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:00.719201 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:55:00.719283 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:00.719265 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:55:00.719318 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:00.719300 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:55:01.723986 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:01.723952 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" event={"ID":"a2300534-a518-4f0c-abb7-5d0e70d301dc","Type":"ContainerStarted","Data":"13a46eed72b930e5fa112dbf3d6af549c98ba5d2916af81375988970f1379edf"} Aug 05 16:55:02.589005 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:02.588963 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:55:02.590047 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:02.590022 2573 patch_prober.go:28] interesting pod/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Aug 05 16:55:02.590125 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:02.590075 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" podUID="a2300534-a518-4f0c-abb7-5d0e70d301dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Aug 05 16:55:02.656032 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:02.655985 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:55:03.589229 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:03.589190 2573 patch_prober.go:28] interesting pod/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Aug 05 16:55:03.589695 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:03.589256 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" podUID="a2300534-a518-4f0c-abb7-5d0e70d301dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Aug 05 16:55:04.588985 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:04.588938 2573 patch_prober.go:28] interesting pod/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Aug 05 16:55:04.589162 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:04.589011 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" podUID="a2300534-a518-4f0c-abb7-5d0e70d301dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Aug 05 16:55:05.588545 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:05.588505 2573 patch_prober.go:28] interesting pod/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Aug 05 16:55:05.588956 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:05.588564 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" podUID="a2300534-a518-4f0c-abb7-5d0e70d301dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Aug 05 16:55:06.589546 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:06.589501 2573 patch_prober.go:28] interesting pod/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Aug 05 16:55:06.589969 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:06.589579 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" podUID="a2300534-a518-4f0c-abb7-5d0e70d301dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Aug 05 16:55:07.592828 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:07.592801 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:55:07.593712 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:07.593691 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:55:29.917873 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:29.917841 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:30.035580 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:30.035547 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:30.153282 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:30.153246 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:30.226875 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:30.226778 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:31.385844 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:31.385813 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt"] Aug 05 16:55:31.386278 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:31.386018 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" podUID="a2300534-a518-4f0c-abb7-5d0e70d301dc" containerName="istio-proxy" containerID="cri-o://13a46eed72b930e5fa112dbf3d6af549c98ba5d2916af81375988970f1379edf" gracePeriod=30 Aug 05 16:55:31.445180 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:31.445144 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:31.625926 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:31.625897 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:31.868076 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:31.868044 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:32.296363 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.296328 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4"] Aug 05 16:55:32.296725 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.296708 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29" containerName="istio-proxy" Aug 05 16:55:32.296811 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.296728 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29" containerName="istio-proxy" Aug 05 16:55:32.296866 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.296825 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29" containerName="istio-proxy" Aug 05 16:55:32.296866 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.296839 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29" containerName="istio-proxy" Aug 05 16:55:32.298784 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.298763 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.302864 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.302837 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65905846-6d59-49eb-8bcf-1a1293246034-istiod-ca-cert\") pod \"e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.302950 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.302902 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm6jf\" (UniqueName: \"kubernetes.io/projected/65905846-6d59-49eb-8bcf-1a1293246034-kube-api-access-tm6jf\") pod \"e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.302950 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.302934 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-istio-envoy\") pod \"e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.303035 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.303007 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-workload-certs\") pod \"e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.303080 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.303034 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65905846-6d59-49eb-8bcf-1a1293246034-istio-token\") pod \"e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.303080 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.303063 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65905846-6d59-49eb-8bcf-1a1293246034-istio-podinfo\") pod \"e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.303161 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.303094 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-workload-socket\") pod \"e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.303161 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.303150 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-credential-socket\") pod \"e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.303254 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.303174 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-istio-data\") pod \"e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.310484 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.310467 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-e051d82f-openshift-default-dockercfg-fm82j\"" Aug 05 16:55:32.330673 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.330649 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4"] Aug 05 16:55:32.365542 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.365504 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:32.403733 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.403701 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-workload-socket\") pod \"e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.404202 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.403757 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-credential-socket\") pod \"e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.404202 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.403785 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-istio-data\") pod \"e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.404202 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.403832 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65905846-6d59-49eb-8bcf-1a1293246034-istiod-ca-cert\") pod \"e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.404202 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.403876 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm6jf\" (UniqueName: \"kubernetes.io/projected/65905846-6d59-49eb-8bcf-1a1293246034-kube-api-access-tm6jf\") pod \"e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.404202 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.403910 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-istio-envoy\") pod \"e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.404202 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.403955 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-workload-certs\") pod \"e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.404202 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.403980 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65905846-6d59-49eb-8bcf-1a1293246034-istio-token\") pod \"e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.404202 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.404005 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65905846-6d59-49eb-8bcf-1a1293246034-istio-podinfo\") pod \"e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.404202 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.404148 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-workload-socket\") pod \"e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.404202 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.404195 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-credential-socket\") pod \"e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.404787 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.404464 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-istio-data\") pod \"e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.404787 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.404700 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-workload-certs\") pod \"e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.405139 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.405107 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65905846-6d59-49eb-8bcf-1a1293246034-istiod-ca-cert\") pod \"e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.407085 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.406953 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65905846-6d59-49eb-8bcf-1a1293246034-istio-podinfo\") pod \"e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.407085 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.407023 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-istio-envoy\") pod \"e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.428272 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.428236 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm6jf\" (UniqueName: \"kubernetes.io/projected/65905846-6d59-49eb-8bcf-1a1293246034-kube-api-access-tm6jf\") pod \"e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.430436 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.430409 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65905846-6d59-49eb-8bcf-1a1293246034-istio-token\") pod \"e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.608607 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.608509 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:32.774337 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.774311 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4"] Aug 05 16:55:32.776223 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:55:32.776196 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65905846_6d59_49eb_8bcf_1a1293246034.slice/crio-4e4ea319be2efbfcbc381c11391bfda499f55c2646cc24aff9d414f0c46e8d2b WatchSource:0}: Error finding container 4e4ea319be2efbfcbc381c11391bfda499f55c2646cc24aff9d414f0c46e8d2b: Status 404 returned error can't find the container with id 4e4ea319be2efbfcbc381c11391bfda499f55c2646cc24aff9d414f0c46e8d2b Aug 05 16:55:32.778281 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.778249 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:55:32.778366 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.778310 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:55:32.778366 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.778342 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:55:32.871412 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.871311 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" event={"ID":"65905846-6d59-49eb-8bcf-1a1293246034","Type":"ContainerStarted","Data":"ec76c4e392f227090c4c38d5f4dccc7822f1c357f96430eeba3486b74ce0c9a2"} Aug 05 16:55:32.871412 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.871369 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" event={"ID":"65905846-6d59-49eb-8bcf-1a1293246034","Type":"ContainerStarted","Data":"4e4ea319be2efbfcbc381c11391bfda499f55c2646cc24aff9d414f0c46e8d2b"} Aug 05 16:55:32.916052 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.915995 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" podStartSLOduration=0.915979025 podStartE2EDuration="915.979025ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:55:32.915845987 +0000 UTC m=+2048.072691887" watchObservedRunningTime="2026-08-05 16:55:32.915979025 +0000 UTC m=+2048.072824925" Aug 05 16:55:32.944181 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:32.944138 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:33.609202 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:33.609170 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:33.610824 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:33.610799 2573 patch_prober.go:28] interesting pod/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Aug 05 16:55:33.610939 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:33.610846 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" podUID="65905846-6d59-49eb-8bcf-1a1293246034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Aug 05 16:55:34.471435 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:34.471368 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:34.582798 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:34.582760 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:34.613608 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:34.613582 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:34.879444 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:34.879410 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:34.880290 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:34.880271 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:55:35.763850 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:35.763817 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:35.899081 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:35.899048 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:36.648346 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.648287 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:55:36.662045 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.662013 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:36.740429 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.740393 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-istio-envoy\") pod \"a2300534-a518-4f0c-abb7-5d0e70d301dc\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " Aug 05 16:55:36.740429 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.740425 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2300534-a518-4f0c-abb7-5d0e70d301dc-istio-token\") pod \"a2300534-a518-4f0c-abb7-5d0e70d301dc\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " Aug 05 16:55:36.740651 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.740442 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-workload-certs\") pod \"a2300534-a518-4f0c-abb7-5d0e70d301dc\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " Aug 05 16:55:36.740651 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.740637 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-workload-socket\") pod \"a2300534-a518-4f0c-abb7-5d0e70d301dc\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " Aug 05 16:55:36.740788 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.740668 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-credential-socket\") pod \"a2300534-a518-4f0c-abb7-5d0e70d301dc\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " Aug 05 16:55:36.740788 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.740715 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5nmq\" (UniqueName: \"kubernetes.io/projected/a2300534-a518-4f0c-abb7-5d0e70d301dc-kube-api-access-n5nmq\") pod \"a2300534-a518-4f0c-abb7-5d0e70d301dc\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " Aug 05 16:55:36.740788 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.740722 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a2300534-a518-4f0c-abb7-5d0e70d301dc" (UID: "a2300534-a518-4f0c-abb7-5d0e70d301dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:55:36.740788 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.740739 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-istio-data\") pod \"a2300534-a518-4f0c-abb7-5d0e70d301dc\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " Aug 05 16:55:36.740995 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.740817 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2300534-a518-4f0c-abb7-5d0e70d301dc-istiod-ca-cert\") pod \"a2300534-a518-4f0c-abb7-5d0e70d301dc\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " Aug 05 16:55:36.740995 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.740850 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2300534-a518-4f0c-abb7-5d0e70d301dc-istio-podinfo\") pod \"a2300534-a518-4f0c-abb7-5d0e70d301dc\" (UID: \"a2300534-a518-4f0c-abb7-5d0e70d301dc\") " Aug 05 16:55:36.740995 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.740920 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a2300534-a518-4f0c-abb7-5d0e70d301dc" (UID: "a2300534-a518-4f0c-abb7-5d0e70d301dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:55:36.740995 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.740930 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a2300534-a518-4f0c-abb7-5d0e70d301dc" (UID: "a2300534-a518-4f0c-abb7-5d0e70d301dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:55:36.741201 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.741143 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-credential-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:55:36.741201 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.741165 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-workload-certs\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:55:36.741201 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.741179 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-workload-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:55:36.741419 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.741386 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-istio-data" (OuterVolumeSpecName: "istio-data") pod "a2300534-a518-4f0c-abb7-5d0e70d301dc" (UID: "a2300534-a518-4f0c-abb7-5d0e70d301dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:55:36.741573 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.741552 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2300534-a518-4f0c-abb7-5d0e70d301dc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a2300534-a518-4f0c-abb7-5d0e70d301dc" (UID: "a2300534-a518-4f0c-abb7-5d0e70d301dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:55:36.742797 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.742771 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a2300534-a518-4f0c-abb7-5d0e70d301dc" (UID: "a2300534-a518-4f0c-abb7-5d0e70d301dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:55:36.742891 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.742834 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2300534-a518-4f0c-abb7-5d0e70d301dc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a2300534-a518-4f0c-abb7-5d0e70d301dc" (UID: "a2300534-a518-4f0c-abb7-5d0e70d301dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 16:55:36.743020 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.742999 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2300534-a518-4f0c-abb7-5d0e70d301dc-istio-token" (OuterVolumeSpecName: "istio-token") pod "a2300534-a518-4f0c-abb7-5d0e70d301dc" (UID: "a2300534-a518-4f0c-abb7-5d0e70d301dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:55:36.743058 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.743007 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2300534-a518-4f0c-abb7-5d0e70d301dc-kube-api-access-n5nmq" (OuterVolumeSpecName: "kube-api-access-n5nmq") pod "a2300534-a518-4f0c-abb7-5d0e70d301dc" (UID: "a2300534-a518-4f0c-abb7-5d0e70d301dc"). InnerVolumeSpecName "kube-api-access-n5nmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:55:36.806043 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.806005 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:55:36.841757 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.841723 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5nmq\" (UniqueName: \"kubernetes.io/projected/a2300534-a518-4f0c-abb7-5d0e70d301dc-kube-api-access-n5nmq\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:55:36.841757 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.841752 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-istio-data\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:55:36.841757 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.841765 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2300534-a518-4f0c-abb7-5d0e70d301dc-istiod-ca-cert\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:55:36.841988 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.841776 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2300534-a518-4f0c-abb7-5d0e70d301dc-istio-podinfo\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:55:36.841988 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.841786 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2300534-a518-4f0c-abb7-5d0e70d301dc-istio-envoy\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:55:36.841988 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.841797 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2300534-a518-4f0c-abb7-5d0e70d301dc-istio-token\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:55:36.890771 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.890727 2573 generic.go:356] "Generic (PLEG): container finished" podID="a2300534-a518-4f0c-abb7-5d0e70d301dc" containerID="13a46eed72b930e5fa112dbf3d6af549c98ba5d2916af81375988970f1379edf" exitCode=0 Aug 05 16:55:36.890978 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.890798 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" Aug 05 16:55:36.890978 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.890886 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" event={"ID":"a2300534-a518-4f0c-abb7-5d0e70d301dc","Type":"ContainerDied","Data":"13a46eed72b930e5fa112dbf3d6af549c98ba5d2916af81375988970f1379edf"} Aug 05 16:55:36.890978 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.890939 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt" event={"ID":"a2300534-a518-4f0c-abb7-5d0e70d301dc","Type":"ContainerDied","Data":"0583b52d073ecc79ce37e3fbb8e07007037ed617892e9d2a61fd003e575140b6"} Aug 05 16:55:36.890978 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.890970 2573 scope.go:117] "RemoveContainer" containerID="13a46eed72b930e5fa112dbf3d6af549c98ba5d2916af81375988970f1379edf" Aug 05 16:55:36.901944 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.901926 2573 scope.go:117] "RemoveContainer" containerID="ff98676de95bf2252eadfeb9cd4c719b0ec5900dc4b815cf211c5d63881f7754" Aug 05 16:55:36.909667 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.909630 2573 scope.go:117] "RemoveContainer" containerID="13a46eed72b930e5fa112dbf3d6af549c98ba5d2916af81375988970f1379edf" Aug 05 16:55:36.909948 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:55:36.909922 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13a46eed72b930e5fa112dbf3d6af549c98ba5d2916af81375988970f1379edf\": container with ID starting with 13a46eed72b930e5fa112dbf3d6af549c98ba5d2916af81375988970f1379edf not found: ID does not exist" containerID="13a46eed72b930e5fa112dbf3d6af549c98ba5d2916af81375988970f1379edf" Aug 05 16:55:36.910033 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.909959 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13a46eed72b930e5fa112dbf3d6af549c98ba5d2916af81375988970f1379edf"} err="failed to get container status \"13a46eed72b930e5fa112dbf3d6af549c98ba5d2916af81375988970f1379edf\": rpc error: code = NotFound desc = could not find container \"13a46eed72b930e5fa112dbf3d6af549c98ba5d2916af81375988970f1379edf\": container with ID starting with 13a46eed72b930e5fa112dbf3d6af549c98ba5d2916af81375988970f1379edf not found: ID does not exist" Aug 05 16:55:36.910033 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.909980 2573 scope.go:117] "RemoveContainer" containerID="ff98676de95bf2252eadfeb9cd4c719b0ec5900dc4b815cf211c5d63881f7754" Aug 05 16:55:36.910228 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:55:36.910213 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff98676de95bf2252eadfeb9cd4c719b0ec5900dc4b815cf211c5d63881f7754\": container with ID starting with ff98676de95bf2252eadfeb9cd4c719b0ec5900dc4b815cf211c5d63881f7754 not found: ID does not exist" containerID="ff98676de95bf2252eadfeb9cd4c719b0ec5900dc4b815cf211c5d63881f7754" Aug 05 16:55:36.910264 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.910230 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff98676de95bf2252eadfeb9cd4c719b0ec5900dc4b815cf211c5d63881f7754"} err="failed to get container status \"ff98676de95bf2252eadfeb9cd4c719b0ec5900dc4b815cf211c5d63881f7754\": rpc error: code = NotFound desc = could not find container \"ff98676de95bf2252eadfeb9cd4c719b0ec5900dc4b815cf211c5d63881f7754\": container with ID starting with ff98676de95bf2252eadfeb9cd4c719b0ec5900dc4b815cf211c5d63881f7754 not found: ID does not exist" Aug 05 16:55:36.934073 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.934041 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt"] Aug 05 16:55:36.941477 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:36.941451 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ff3f79bc-openshift-default-648767fd7b-v2hxt"] Aug 05 16:55:37.304001 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:37.303968 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2300534-a518-4f0c-abb7-5d0e70d301dc" path="/var/lib/kubelet/pods/a2300534-a518-4f0c-abb7-5d0e70d301dc/volumes" Aug 05 16:55:59.170626 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:59.170543 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e051d82f-8wstr"] Aug 05 16:55:59.171012 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:59.170880 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29" containerName="istio-proxy" Aug 05 16:55:59.171012 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:59.170891 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="caeb09f4-5bad-4a7e-87c7-2cda2a2f2e29" containerName="istio-proxy" Aug 05 16:55:59.171012 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:59.170898 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2300534-a518-4f0c-abb7-5d0e70d301dc" containerName="istio-proxy" Aug 05 16:55:59.171012 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:59.170904 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2300534-a518-4f0c-abb7-5d0e70d301dc" containerName="istio-proxy" Aug 05 16:55:59.171012 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:59.170919 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2300534-a518-4f0c-abb7-5d0e70d301dc" containerName="istio-proxy" Aug 05 16:55:59.171012 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:59.170924 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2300534-a518-4f0c-abb7-5d0e70d301dc" containerName="istio-proxy" Aug 05 16:55:59.171012 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:59.170969 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2300534-a518-4f0c-abb7-5d0e70d301dc" containerName="istio-proxy" Aug 05 16:55:59.171012 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:59.170978 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2300534-a518-4f0c-abb7-5d0e70d301dc" containerName="istio-proxy" Aug 05 16:55:59.173907 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:59.173887 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e051d82f-8wstr" Aug 05 16:55:59.177326 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:59.177295 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 05 16:55:59.177326 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:59.177323 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q6h9g\"" Aug 05 16:55:59.184828 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:59.184800 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e051d82f-8wstr"] Aug 05 16:55:59.227629 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:59.227601 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bc672bd-c61d-413c-9a77-db754cfb5588-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e051d82f-8wstr\" (UID: \"6bc672bd-c61d-413c-9a77-db754cfb5588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e051d82f-8wstr" Aug 05 16:55:59.328438 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:59.328405 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bc672bd-c61d-413c-9a77-db754cfb5588-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e051d82f-8wstr\" (UID: \"6bc672bd-c61d-413c-9a77-db754cfb5588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e051d82f-8wstr" Aug 05 16:55:59.329046 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:59.329026 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bc672bd-c61d-413c-9a77-db754cfb5588-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e051d82f-8wstr\" (UID: \"6bc672bd-c61d-413c-9a77-db754cfb5588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e051d82f-8wstr" Aug 05 16:55:59.485055 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:59.484962 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e051d82f-8wstr" Aug 05 16:55:59.627353 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:59.627312 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e051d82f-8wstr"] Aug 05 16:55:59.630162 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:55:59.630125 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc672bd_c61d_413c_9a77_db754cfb5588.slice/crio-b4fe573a689a16c01f244876bacd5542c9107fe7eb631b783029581bb6ba40dd WatchSource:0}: Error finding container b4fe573a689a16c01f244876bacd5542c9107fe7eb631b783029581bb6ba40dd: Status 404 returned error can't find the container with id b4fe573a689a16c01f244876bacd5542c9107fe7eb631b783029581bb6ba40dd Aug 05 16:55:59.998113 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:59.998027 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e051d82f-8wstr" event={"ID":"6bc672bd-c61d-413c-9a77-db754cfb5588","Type":"ContainerStarted","Data":"f9a6b9f4884f03d5f89a7e5cf8c912df810c1a218c9e2687803261aae870b18d"} Aug 05 16:55:59.998113 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:55:59.998081 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e051d82f-8wstr" event={"ID":"6bc672bd-c61d-413c-9a77-db754cfb5588","Type":"ContainerStarted","Data":"b4fe573a689a16c01f244876bacd5542c9107fe7eb631b783029581bb6ba40dd"} Aug 05 16:56:00.021568 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:00.021483 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e051d82f-8wstr" podStartSLOduration=1.021465254 podStartE2EDuration="1.021465254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:56:00.020826163 +0000 UTC m=+2075.177672063" watchObservedRunningTime="2026-08-05 16:56:00.021465254 +0000 UTC m=+2075.178311154" Aug 05 16:56:01.004741 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:01.004701 2573 generic.go:356] "Generic (PLEG): container finished" podID="6bc672bd-c61d-413c-9a77-db754cfb5588" containerID="f9a6b9f4884f03d5f89a7e5cf8c912df810c1a218c9e2687803261aae870b18d" exitCode=0 Aug 05 16:56:01.005112 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:01.004763 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e051d82f-8wstr" event={"ID":"6bc672bd-c61d-413c-9a77-db754cfb5588","Type":"ContainerDied","Data":"f9a6b9f4884f03d5f89a7e5cf8c912df810c1a218c9e2687803261aae870b18d"} Aug 05 16:56:02.137320 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:02.137296 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e051d82f-8wstr" Aug 05 16:56:02.253600 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:02.253565 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bc672bd-c61d-413c-9a77-db754cfb5588-maas-api-service-ca\") pod \"6bc672bd-c61d-413c-9a77-db754cfb5588\" (UID: \"6bc672bd-c61d-413c-9a77-db754cfb5588\") " Aug 05 16:56:02.253962 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:02.253938 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bc672bd-c61d-413c-9a77-db754cfb5588-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6bc672bd-c61d-413c-9a77-db754cfb5588" (UID: "6bc672bd-c61d-413c-9a77-db754cfb5588"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:56:02.355103 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:02.355021 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bc672bd-c61d-413c-9a77-db754cfb5588-maas-api-service-ca\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:56:03.015162 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:03.015122 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e051d82f-8wstr" event={"ID":"6bc672bd-c61d-413c-9a77-db754cfb5588","Type":"ContainerDied","Data":"b4fe573a689a16c01f244876bacd5542c9107fe7eb631b783029581bb6ba40dd"} Aug 05 16:56:03.015162 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:03.015162 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e051d82f-8wstr" Aug 05 16:56:03.015455 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:03.015177 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4fe573a689a16c01f244876bacd5542c9107fe7eb631b783029581bb6ba40dd" Aug 05 16:56:06.599261 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:06.599225 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:06.789179 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:06.789144 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:06.979627 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:06.979535 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:06.997753 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:06.997720 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e051d82f-8wstr"] Aug 05 16:56:07.004356 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:07.004330 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e051d82f-8wstr"] Aug 05 16:56:07.115360 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:07.115323 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:07.180724 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:07.180695 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4"] Aug 05 16:56:07.180905 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:07.180885 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" podUID="65905846-6d59-49eb-8bcf-1a1293246034" containerName="istio-proxy" containerID="cri-o://ec76c4e392f227090c4c38d5f4dccc7822f1c357f96430eeba3486b74ce0c9a2" gracePeriod=30 Aug 05 16:56:07.252352 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:07.252240 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:07.303282 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:07.303251 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bc672bd-c61d-413c-9a77-db754cfb5588" path="/var/lib/kubelet/pods/6bc672bd-c61d-413c-9a77-db754cfb5588/volumes" Aug 05 16:56:07.442600 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:07.442564 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:08.033447 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:08.033409 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:08.270466 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:08.270435 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:15.726048 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:15.726023 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:56:15.877469 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:15.877364 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65905846-6d59-49eb-8bcf-1a1293246034-istio-podinfo\") pod \"65905846-6d59-49eb-8bcf-1a1293246034\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " Aug 05 16:56:15.877469 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:15.877449 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-istio-envoy\") pod \"65905846-6d59-49eb-8bcf-1a1293246034\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " Aug 05 16:56:15.877709 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:15.877480 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-credential-socket\") pod \"65905846-6d59-49eb-8bcf-1a1293246034\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " Aug 05 16:56:15.877709 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:15.877530 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65905846-6d59-49eb-8bcf-1a1293246034-istiod-ca-cert\") pod \"65905846-6d59-49eb-8bcf-1a1293246034\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " Aug 05 16:56:15.877709 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:15.877551 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-workload-socket\") pod \"65905846-6d59-49eb-8bcf-1a1293246034\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " Aug 05 16:56:15.877709 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:15.877575 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-workload-certs\") pod \"65905846-6d59-49eb-8bcf-1a1293246034\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " Aug 05 16:56:15.877709 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:15.877616 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tm6jf\" (UniqueName: \"kubernetes.io/projected/65905846-6d59-49eb-8bcf-1a1293246034-kube-api-access-tm6jf\") pod \"65905846-6d59-49eb-8bcf-1a1293246034\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " Aug 05 16:56:15.877709 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:15.877638 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-istio-data\") pod \"65905846-6d59-49eb-8bcf-1a1293246034\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " Aug 05 16:56:15.877709 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:15.877670 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65905846-6d59-49eb-8bcf-1a1293246034-istio-token\") pod \"65905846-6d59-49eb-8bcf-1a1293246034\" (UID: \"65905846-6d59-49eb-8bcf-1a1293246034\") " Aug 05 16:56:15.878068 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:15.877852 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "65905846-6d59-49eb-8bcf-1a1293246034" (UID: "65905846-6d59-49eb-8bcf-1a1293246034"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:56:15.878068 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:15.877955 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-workload-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:56:15.878068 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:15.877956 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65905846-6d59-49eb-8bcf-1a1293246034-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "65905846-6d59-49eb-8bcf-1a1293246034" (UID: "65905846-6d59-49eb-8bcf-1a1293246034"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:56:15.878068 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:15.877858 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "65905846-6d59-49eb-8bcf-1a1293246034" (UID: "65905846-6d59-49eb-8bcf-1a1293246034"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:56:15.878068 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:15.878045 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-istio-data" (OuterVolumeSpecName: "istio-data") pod "65905846-6d59-49eb-8bcf-1a1293246034" (UID: "65905846-6d59-49eb-8bcf-1a1293246034"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:56:15.878271 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:15.878067 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "65905846-6d59-49eb-8bcf-1a1293246034" (UID: "65905846-6d59-49eb-8bcf-1a1293246034"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:56:15.879917 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:15.879889 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65905846-6d59-49eb-8bcf-1a1293246034-istio-token" (OuterVolumeSpecName: "istio-token") pod "65905846-6d59-49eb-8bcf-1a1293246034" (UID: "65905846-6d59-49eb-8bcf-1a1293246034"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:56:15.880045 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:15.879949 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65905846-6d59-49eb-8bcf-1a1293246034-kube-api-access-tm6jf" (OuterVolumeSpecName: "kube-api-access-tm6jf") pod "65905846-6d59-49eb-8bcf-1a1293246034" (UID: "65905846-6d59-49eb-8bcf-1a1293246034"). InnerVolumeSpecName "kube-api-access-tm6jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:56:15.880126 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:15.880084 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "65905846-6d59-49eb-8bcf-1a1293246034" (UID: "65905846-6d59-49eb-8bcf-1a1293246034"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:56:15.880126 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:15.880103 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65905846-6d59-49eb-8bcf-1a1293246034-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "65905846-6d59-49eb-8bcf-1a1293246034" (UID: "65905846-6d59-49eb-8bcf-1a1293246034"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 16:56:15.978527 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:15.978483 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-istio-envoy\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:56:15.978527 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:15.978523 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-credential-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:56:15.978744 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:15.978538 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65905846-6d59-49eb-8bcf-1a1293246034-istiod-ca-cert\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:56:15.978744 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:15.978552 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-workload-certs\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:56:15.978744 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:15.978567 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm6jf\" (UniqueName: \"kubernetes.io/projected/65905846-6d59-49eb-8bcf-1a1293246034-kube-api-access-tm6jf\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:56:15.978744 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:15.978580 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65905846-6d59-49eb-8bcf-1a1293246034-istio-data\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:56:15.978744 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:15.978594 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65905846-6d59-49eb-8bcf-1a1293246034-istio-token\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:56:15.978744 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:15.978607 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65905846-6d59-49eb-8bcf-1a1293246034-istio-podinfo\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:56:16.088172 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:16.088142 2573 generic.go:356] "Generic (PLEG): container finished" podID="65905846-6d59-49eb-8bcf-1a1293246034" containerID="ec76c4e392f227090c4c38d5f4dccc7822f1c357f96430eeba3486b74ce0c9a2" exitCode=0 Aug 05 16:56:16.088356 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:16.088189 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" Aug 05 16:56:16.088356 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:16.088197 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" event={"ID":"65905846-6d59-49eb-8bcf-1a1293246034","Type":"ContainerDied","Data":"ec76c4e392f227090c4c38d5f4dccc7822f1c357f96430eeba3486b74ce0c9a2"} Aug 05 16:56:16.088356 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:16.088248 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4" event={"ID":"65905846-6d59-49eb-8bcf-1a1293246034","Type":"ContainerDied","Data":"4e4ea319be2efbfcbc381c11391bfda499f55c2646cc24aff9d414f0c46e8d2b"} Aug 05 16:56:16.088356 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:16.088277 2573 scope.go:117] "RemoveContainer" containerID="ec76c4e392f227090c4c38d5f4dccc7822f1c357f96430eeba3486b74ce0c9a2" Aug 05 16:56:16.097488 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:16.097467 2573 scope.go:117] "RemoveContainer" containerID="ec76c4e392f227090c4c38d5f4dccc7822f1c357f96430eeba3486b74ce0c9a2" Aug 05 16:56:16.097770 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:56:16.097751 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec76c4e392f227090c4c38d5f4dccc7822f1c357f96430eeba3486b74ce0c9a2\": container with ID starting with ec76c4e392f227090c4c38d5f4dccc7822f1c357f96430eeba3486b74ce0c9a2 not found: ID does not exist" containerID="ec76c4e392f227090c4c38d5f4dccc7822f1c357f96430eeba3486b74ce0c9a2" Aug 05 16:56:16.097833 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:16.097778 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec76c4e392f227090c4c38d5f4dccc7822f1c357f96430eeba3486b74ce0c9a2"} err="failed to get container status \"ec76c4e392f227090c4c38d5f4dccc7822f1c357f96430eeba3486b74ce0c9a2\": rpc error: code = NotFound desc = could not find container \"ec76c4e392f227090c4c38d5f4dccc7822f1c357f96430eeba3486b74ce0c9a2\": container with ID starting with ec76c4e392f227090c4c38d5f4dccc7822f1c357f96430eeba3486b74ce0c9a2 not found: ID does not exist" Aug 05 16:56:16.113038 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:16.113013 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4"] Aug 05 16:56:16.120421 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:16.120400 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-e051d82f-openshift-default-7c5bd659d4-zxlh4"] Aug 05 16:56:17.303627 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:17.303591 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65905846-6d59-49eb-8bcf-1a1293246034" path="/var/lib/kubelet/pods/65905846-6d59-49eb-8bcf-1a1293246034/volumes" Aug 05 16:56:19.942622 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:19.942587 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:26.850634 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:26.850601 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:26.953835 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:26.953793 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:27.059095 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:27.059061 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:27.531912 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:27.531883 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:27.633240 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:27.633204 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:53.494445 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:53.494412 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:59.880266 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:59.880230 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:56:59.984679 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:56:59.984649 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:57:00.086605 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:57:00.086571 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:57:00.190036 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:57:00.189948 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:57:01.838128 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:57:01.838098 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:57:15.951548 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:57:15.951514 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:57:22.055214 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:57:22.055117 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:57:22.180631 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:57:22.180598 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:57:22.376673 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:57:22.376578 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:57:32.769876 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:57:32.769845 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:57:43.779409 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:57:43.779358 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:57:50.187386 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:57:50.187352 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:57:50.306243 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:57:50.306210 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:57:50.432179 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:57:50.432147 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:57:50.606220 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:57:50.606185 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:57:51.239818 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:57:51.239788 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:57:51.345790 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:57:51.345758 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:57:52.910639 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:57:52.910600 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:57:59.231869 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:57:59.231835 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:57:59.340158 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:57:59.340117 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:57:59.920953 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:57:59.920923 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:04.026220 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:04.026189 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:04.213567 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:04.213528 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:06.888107 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:06.888070 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:06.987946 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:06.987910 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:07.141352 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.141255 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:07.513248 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.513210 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc"] Aug 05 16:58:07.513660 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.513635 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bc672bd-c61d-413c-9a77-db754cfb5588" containerName="revoke-keys" Aug 05 16:58:07.513660 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.513660 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bc672bd-c61d-413c-9a77-db754cfb5588" containerName="revoke-keys" Aug 05 16:58:07.513907 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.513683 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65905846-6d59-49eb-8bcf-1a1293246034" containerName="istio-proxy" Aug 05 16:58:07.513907 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.513692 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="65905846-6d59-49eb-8bcf-1a1293246034" containerName="istio-proxy" Aug 05 16:58:07.513907 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.513803 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="65905846-6d59-49eb-8bcf-1a1293246034" containerName="istio-proxy" Aug 05 16:58:07.513907 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.513821 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bc672bd-c61d-413c-9a77-db754cfb5588" containerName="revoke-keys" Aug 05 16:58:07.516610 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.516588 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.521253 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.521230 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-29433603-openshift-default-dockercfg-wchsc\"" Aug 05 16:58:07.535099 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.535071 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:07.546752 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.546722 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc"] Aug 05 16:58:07.631349 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.631311 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istio-podinfo\") pod \"e2e-mt-29433603-openshift-default-797d58c956-6d7jc\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.631566 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.631358 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istio-token\") pod \"e2e-mt-29433603-openshift-default-797d58c956-6d7jc\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.631566 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.631452 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-workload-socket\") pod \"e2e-mt-29433603-openshift-default-797d58c956-6d7jc\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.631566 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.631501 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-workload-certs\") pod \"e2e-mt-29433603-openshift-default-797d58c956-6d7jc\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.631566 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.631559 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-credential-socket\") pod \"e2e-mt-29433603-openshift-default-797d58c956-6d7jc\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.631783 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.631605 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istio-envoy\") pod \"e2e-mt-29433603-openshift-default-797d58c956-6d7jc\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.631783 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.631633 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istio-data\") pod \"e2e-mt-29433603-openshift-default-797d58c956-6d7jc\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.631783 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.631656 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istiod-ca-cert\") pod \"e2e-mt-29433603-openshift-default-797d58c956-6d7jc\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.631783 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.631685 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfpdg\" (UniqueName: \"kubernetes.io/projected/838cbb08-2f8f-4e7c-a74c-7d2e17928115-kube-api-access-bfpdg\") pod \"e2e-mt-29433603-openshift-default-797d58c956-6d7jc\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.709659 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.709618 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:07.733027 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.732997 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-workload-socket\") pod \"e2e-mt-29433603-openshift-default-797d58c956-6d7jc\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.733170 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.733033 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-workload-certs\") pod \"e2e-mt-29433603-openshift-default-797d58c956-6d7jc\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.733170 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.733061 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-credential-socket\") pod \"e2e-mt-29433603-openshift-default-797d58c956-6d7jc\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.733170 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.733098 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istio-envoy\") pod \"e2e-mt-29433603-openshift-default-797d58c956-6d7jc\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.733170 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.733119 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istio-data\") pod \"e2e-mt-29433603-openshift-default-797d58c956-6d7jc\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.733170 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.733142 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istiod-ca-cert\") pod \"e2e-mt-29433603-openshift-default-797d58c956-6d7jc\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.733170 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.733168 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfpdg\" (UniqueName: \"kubernetes.io/projected/838cbb08-2f8f-4e7c-a74c-7d2e17928115-kube-api-access-bfpdg\") pod \"e2e-mt-29433603-openshift-default-797d58c956-6d7jc\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.733450 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.733238 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istio-podinfo\") pod \"e2e-mt-29433603-openshift-default-797d58c956-6d7jc\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.733450 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.733262 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istio-token\") pod \"e2e-mt-29433603-openshift-default-797d58c956-6d7jc\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.733563 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.733490 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-credential-socket\") pod \"e2e-mt-29433603-openshift-default-797d58c956-6d7jc\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.733563 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.733515 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-workload-socket\") pod \"e2e-mt-29433603-openshift-default-797d58c956-6d7jc\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.733738 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.733713 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istio-data\") pod \"e2e-mt-29433603-openshift-default-797d58c956-6d7jc\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.733833 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.733766 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-workload-certs\") pod \"e2e-mt-29433603-openshift-default-797d58c956-6d7jc\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.734149 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.734128 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istiod-ca-cert\") pod \"e2e-mt-29433603-openshift-default-797d58c956-6d7jc\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.735716 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.735680 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istio-envoy\") pod \"e2e-mt-29433603-openshift-default-797d58c956-6d7jc\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.735987 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.735966 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istio-podinfo\") pod \"e2e-mt-29433603-openshift-default-797d58c956-6d7jc\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.743929 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.743907 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istio-token\") pod \"e2e-mt-29433603-openshift-default-797d58c956-6d7jc\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.744093 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.744073 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfpdg\" (UniqueName: \"kubernetes.io/projected/838cbb08-2f8f-4e7c-a74c-7d2e17928115-kube-api-access-bfpdg\") pod \"e2e-mt-29433603-openshift-default-797d58c956-6d7jc\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.827987 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.827890 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:07.963026 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.962970 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc"] Aug 05 16:58:07.965109 ip-10-0-132-37 kubenswrapper[2573]: W0805 16:58:07.965080 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838cbb08_2f8f_4e7c_a74c_7d2e17928115.slice/crio-53cef4cb22382a5619ab5e1cb757b5730f26c22ed8391c7941062d5a83a7e40f WatchSource:0}: Error finding container 53cef4cb22382a5619ab5e1cb757b5730f26c22ed8391c7941062d5a83a7e40f: Status 404 returned error can't find the container with id 53cef4cb22382a5619ab5e1cb757b5730f26c22ed8391c7941062d5a83a7e40f Aug 05 16:58:07.967108 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.967081 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:58:07.967166 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.967146 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:58:07.967201 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:07.967173 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:58:08.603606 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:08.603571 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" event={"ID":"838cbb08-2f8f-4e7c-a74c-7d2e17928115","Type":"ContainerStarted","Data":"8a3480632b1cf41c1623a6825e48b1a1174b3f9f26038cd9d3859b29745258b6"} Aug 05 16:58:08.603787 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:08.603648 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" event={"ID":"838cbb08-2f8f-4e7c-a74c-7d2e17928115","Type":"ContainerStarted","Data":"53cef4cb22382a5619ab5e1cb757b5730f26c22ed8391c7941062d5a83a7e40f"} Aug 05 16:58:08.649057 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:08.649009 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" podStartSLOduration=1.6489948129999998 podStartE2EDuration="1.648994813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 16:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 16:58:08.648555496 +0000 UTC m=+2203.805401433" watchObservedRunningTime="2026-08-05 16:58:08.648994813 +0000 UTC m=+2203.805841118" Aug 05 16:58:08.828456 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:08.828421 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:09.833437 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:09.833398 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:10.613041 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:10.613012 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:10.614138 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:10.614119 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:10.748810 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:10.748779 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:10.912731 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:10.912650 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:11.822568 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:11.822527 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:12.033617 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:12.033576 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:12.583754 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:12.583724 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:12.677655 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:12.677612 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:23.682778 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:23.682745 2573 generic.go:356] "Generic (PLEG): container finished" podID="838cbb08-2f8f-4e7c-a74c-7d2e17928115" containerID="8a3480632b1cf41c1623a6825e48b1a1174b3f9f26038cd9d3859b29745258b6" exitCode=0 Aug 05 16:58:23.683156 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:23.682820 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" event={"ID":"838cbb08-2f8f-4e7c-a74c-7d2e17928115","Type":"ContainerDied","Data":"8a3480632b1cf41c1623a6825e48b1a1174b3f9f26038cd9d3859b29745258b6"} Aug 05 16:58:23.683223 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:23.683210 2573 scope.go:117] "RemoveContainer" containerID="8a3480632b1cf41c1623a6825e48b1a1174b3f9f26038cd9d3859b29745258b6" Aug 05 16:58:23.684150 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:23.684128 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:58:23.684208 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:23.684194 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:58:23.684240 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:23.684224 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 16:58:24.689248 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:24.689210 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" event={"ID":"838cbb08-2f8f-4e7c-a74c-7d2e17928115","Type":"ContainerStarted","Data":"756ba1088831c1fd4a8767294e4c714cea1cea15b7c1be65607923ab42247dee"} Aug 05 16:58:24.828832 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:24.828743 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:24.830147 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:24.829821 2573 patch_prober.go:28] interesting pod/e2e-mt-29433603-openshift-default-797d58c956-6d7jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Aug 05 16:58:24.830147 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:24.829875 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" podUID="838cbb08-2f8f-4e7c-a74c-7d2e17928115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Aug 05 16:58:25.613889 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:25.613856 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:25.829247 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:25.829209 2573 patch_prober.go:28] interesting pod/e2e-mt-29433603-openshift-default-797d58c956-6d7jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Aug 05 16:58:25.829831 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:25.829798 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" podUID="838cbb08-2f8f-4e7c-a74c-7d2e17928115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Aug 05 16:58:26.828614 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:26.828569 2573 patch_prober.go:28] interesting pod/e2e-mt-29433603-openshift-default-797d58c956-6d7jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Aug 05 16:58:26.828838 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:26.828644 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" podUID="838cbb08-2f8f-4e7c-a74c-7d2e17928115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Aug 05 16:58:27.829343 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:27.829299 2573 patch_prober.go:28] interesting pod/e2e-mt-29433603-openshift-default-797d58c956-6d7jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Aug 05 16:58:27.829979 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:27.829942 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" podUID="838cbb08-2f8f-4e7c-a74c-7d2e17928115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Aug 05 16:58:28.828800 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:28.828754 2573 patch_prober.go:28] interesting pod/e2e-mt-29433603-openshift-default-797d58c956-6d7jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Aug 05 16:58:28.829033 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:28.828830 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" podUID="838cbb08-2f8f-4e7c-a74c-7d2e17928115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Aug 05 16:58:29.832386 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:29.832353 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:29.833417 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:29.833394 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:58:35.543933 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:35.543885 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:36.208748 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:36.208718 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:36.300386 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:36.300352 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:36.610166 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:36.610138 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:42.705173 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:42.705132 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:42.787530 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:42.787496 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:53.724676 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:53.724576 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:53.809423 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:53.809389 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:53.906561 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:53.906521 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:54.000309 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:54.000210 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:58:54.356230 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:58:54.356194 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:00.576199 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:00.576158 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:00.643036 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:00.643005 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:00.847077 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:00.846968 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc"] Aug 05 16:59:00.847276 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:00.847188 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" podUID="838cbb08-2f8f-4e7c-a74c-7d2e17928115" containerName="istio-proxy" containerID="cri-o://756ba1088831c1fd4a8767294e4c714cea1cea15b7c1be65607923ab42247dee" gracePeriod=30 Aug 05 16:59:00.882844 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:00.882806 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:01.105415 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:01.105296 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:01.234868 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:01.234818 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:01.332273 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:01.332245 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:01.445749 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:01.445659 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:06.104614 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.104584 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:59:06.231700 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.231622 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istio-podinfo\") pod \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " Aug 05 16:59:06.231700 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.231660 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-credential-socket\") pod \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " Aug 05 16:59:06.231700 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.231684 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-workload-certs\") pod \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " Aug 05 16:59:06.231959 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.231713 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfpdg\" (UniqueName: \"kubernetes.io/projected/838cbb08-2f8f-4e7c-a74c-7d2e17928115-kube-api-access-bfpdg\") pod \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " Aug 05 16:59:06.231959 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.231755 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-workload-socket\") pod \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " Aug 05 16:59:06.231959 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.231810 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istio-token\") pod \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " Aug 05 16:59:06.231959 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.231839 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istio-envoy\") pod \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " Aug 05 16:59:06.231959 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.231870 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istio-data\") pod \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " Aug 05 16:59:06.231959 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.231897 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istiod-ca-cert\") pod \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\" (UID: \"838cbb08-2f8f-4e7c-a74c-7d2e17928115\") " Aug 05 16:59:06.232259 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.231969 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "838cbb08-2f8f-4e7c-a74c-7d2e17928115" (UID: "838cbb08-2f8f-4e7c-a74c-7d2e17928115"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:59:06.232259 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.231992 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "838cbb08-2f8f-4e7c-a74c-7d2e17928115" (UID: "838cbb08-2f8f-4e7c-a74c-7d2e17928115"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:59:06.232259 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.232119 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "838cbb08-2f8f-4e7c-a74c-7d2e17928115" (UID: "838cbb08-2f8f-4e7c-a74c-7d2e17928115"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:59:06.232259 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.232150 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istio-data" (OuterVolumeSpecName: "istio-data") pod "838cbb08-2f8f-4e7c-a74c-7d2e17928115" (UID: "838cbb08-2f8f-4e7c-a74c-7d2e17928115"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:59:06.232259 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.232210 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-credential-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:59:06.232259 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.232229 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-workload-certs\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:59:06.232259 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.232243 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-workload-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:59:06.232259 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.232260 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istio-data\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:59:06.232630 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.232270 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "838cbb08-2f8f-4e7c-a74c-7d2e17928115" (UID: "838cbb08-2f8f-4e7c-a74c-7d2e17928115"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 16:59:06.233890 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.233859 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "838cbb08-2f8f-4e7c-a74c-7d2e17928115" (UID: "838cbb08-2f8f-4e7c-a74c-7d2e17928115"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 16:59:06.234019 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.233965 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istio-token" (OuterVolumeSpecName: "istio-token") pod "838cbb08-2f8f-4e7c-a74c-7d2e17928115" (UID: "838cbb08-2f8f-4e7c-a74c-7d2e17928115"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:59:06.234019 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.233979 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/838cbb08-2f8f-4e7c-a74c-7d2e17928115-kube-api-access-bfpdg" (OuterVolumeSpecName: "kube-api-access-bfpdg") pod "838cbb08-2f8f-4e7c-a74c-7d2e17928115" (UID: "838cbb08-2f8f-4e7c-a74c-7d2e17928115"). InnerVolumeSpecName "kube-api-access-bfpdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:59:06.234112 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.234093 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "838cbb08-2f8f-4e7c-a74c-7d2e17928115" (UID: "838cbb08-2f8f-4e7c-a74c-7d2e17928115"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:59:06.333540 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.333505 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istio-podinfo\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:59:06.333540 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.333536 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfpdg\" (UniqueName: \"kubernetes.io/projected/838cbb08-2f8f-4e7c-a74c-7d2e17928115-kube-api-access-bfpdg\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:59:06.333540 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.333547 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istio-token\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:59:06.333776 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.333555 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istio-envoy\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:59:06.333776 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.333563 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/838cbb08-2f8f-4e7c-a74c-7d2e17928115-istiod-ca-cert\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:59:06.893699 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.893663 2573 generic.go:356] "Generic (PLEG): container finished" podID="838cbb08-2f8f-4e7c-a74c-7d2e17928115" containerID="756ba1088831c1fd4a8767294e4c714cea1cea15b7c1be65607923ab42247dee" exitCode=0 Aug 05 16:59:06.893876 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.893715 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" Aug 05 16:59:06.893876 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.893753 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" event={"ID":"838cbb08-2f8f-4e7c-a74c-7d2e17928115","Type":"ContainerDied","Data":"756ba1088831c1fd4a8767294e4c714cea1cea15b7c1be65607923ab42247dee"} Aug 05 16:59:06.893876 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.893807 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc" event={"ID":"838cbb08-2f8f-4e7c-a74c-7d2e17928115","Type":"ContainerDied","Data":"53cef4cb22382a5619ab5e1cb757b5730f26c22ed8391c7941062d5a83a7e40f"} Aug 05 16:59:06.893876 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.893830 2573 scope.go:117] "RemoveContainer" containerID="756ba1088831c1fd4a8767294e4c714cea1cea15b7c1be65607923ab42247dee" Aug 05 16:59:06.903209 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.903192 2573 scope.go:117] "RemoveContainer" containerID="8a3480632b1cf41c1623a6825e48b1a1174b3f9f26038cd9d3859b29745258b6" Aug 05 16:59:06.911353 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.911333 2573 scope.go:117] "RemoveContainer" containerID="756ba1088831c1fd4a8767294e4c714cea1cea15b7c1be65607923ab42247dee" Aug 05 16:59:06.911645 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:59:06.911625 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"756ba1088831c1fd4a8767294e4c714cea1cea15b7c1be65607923ab42247dee\": container with ID starting with 756ba1088831c1fd4a8767294e4c714cea1cea15b7c1be65607923ab42247dee not found: ID does not exist" containerID="756ba1088831c1fd4a8767294e4c714cea1cea15b7c1be65607923ab42247dee" Aug 05 16:59:06.911683 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.911654 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"756ba1088831c1fd4a8767294e4c714cea1cea15b7c1be65607923ab42247dee"} err="failed to get container status \"756ba1088831c1fd4a8767294e4c714cea1cea15b7c1be65607923ab42247dee\": rpc error: code = NotFound desc = could not find container \"756ba1088831c1fd4a8767294e4c714cea1cea15b7c1be65607923ab42247dee\": container with ID starting with 756ba1088831c1fd4a8767294e4c714cea1cea15b7c1be65607923ab42247dee not found: ID does not exist" Aug 05 16:59:06.911683 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.911674 2573 scope.go:117] "RemoveContainer" containerID="8a3480632b1cf41c1623a6825e48b1a1174b3f9f26038cd9d3859b29745258b6" Aug 05 16:59:06.911952 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:59:06.911936 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a3480632b1cf41c1623a6825e48b1a1174b3f9f26038cd9d3859b29745258b6\": container with ID starting with 8a3480632b1cf41c1623a6825e48b1a1174b3f9f26038cd9d3859b29745258b6 not found: ID does not exist" containerID="8a3480632b1cf41c1623a6825e48b1a1174b3f9f26038cd9d3859b29745258b6" Aug 05 16:59:06.911992 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.911961 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a3480632b1cf41c1623a6825e48b1a1174b3f9f26038cd9d3859b29745258b6"} err="failed to get container status \"8a3480632b1cf41c1623a6825e48b1a1174b3f9f26038cd9d3859b29745258b6\": rpc error: code = NotFound desc = could not find container \"8a3480632b1cf41c1623a6825e48b1a1174b3f9f26038cd9d3859b29745258b6\": container with ID starting with 8a3480632b1cf41c1623a6825e48b1a1174b3f9f26038cd9d3859b29745258b6 not found: ID does not exist" Aug 05 16:59:06.918028 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.918004 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc"] Aug 05 16:59:06.924574 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:06.924553 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-29433603-openshift-default-797d58c956-6d7jc"] Aug 05 16:59:07.304630 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:07.304597 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="838cbb08-2f8f-4e7c-a74c-7d2e17928115" path="/var/lib/kubelet/pods/838cbb08-2f8f-4e7c-a74c-7d2e17928115/volumes" Aug 05 16:59:07.326049 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:07.326020 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:07.436481 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:07.436454 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:07.527188 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:07.527158 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:07.689361 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:07.689278 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:13.388454 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:13.388419 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:13.490654 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:13.490622 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:38.618633 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:38.618599 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 16:59:41.648582 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:41.648546 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-l2cfn"] Aug 05 16:59:41.649051 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:41.648785 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-l2cfn" podUID="3085d0d5-fb73-4f02-9dca-cdb3cf369470" containerName="manager" containerID="cri-o://e3bb466a70540dba24be89971523b78dbcf64a444eec4af9c6965065fddf04e6" gracePeriod=10 Aug 05 16:59:41.899201 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:41.899141 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-l2cfn" Aug 05 16:59:42.040606 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:42.040573 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3085d0d5-fb73-4f02-9dca-cdb3cf369470-extensions-socket-volume\") pod \"3085d0d5-fb73-4f02-9dca-cdb3cf369470\" (UID: \"3085d0d5-fb73-4f02-9dca-cdb3cf369470\") " Aug 05 16:59:42.040785 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:42.040614 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkn7r\" (UniqueName: \"kubernetes.io/projected/3085d0d5-fb73-4f02-9dca-cdb3cf369470-kube-api-access-bkn7r\") pod \"3085d0d5-fb73-4f02-9dca-cdb3cf369470\" (UID: \"3085d0d5-fb73-4f02-9dca-cdb3cf369470\") " Aug 05 16:59:42.040976 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:42.040951 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3085d0d5-fb73-4f02-9dca-cdb3cf369470-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3085d0d5-fb73-4f02-9dca-cdb3cf369470" (UID: "3085d0d5-fb73-4f02-9dca-cdb3cf369470"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 16:59:42.042790 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:42.042761 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3085d0d5-fb73-4f02-9dca-cdb3cf369470-kube-api-access-bkn7r" (OuterVolumeSpecName: "kube-api-access-bkn7r") pod "3085d0d5-fb73-4f02-9dca-cdb3cf369470" (UID: "3085d0d5-fb73-4f02-9dca-cdb3cf369470"). InnerVolumeSpecName "kube-api-access-bkn7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 16:59:42.050184 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:42.050160 2573 generic.go:356] "Generic (PLEG): container finished" podID="3085d0d5-fb73-4f02-9dca-cdb3cf369470" containerID="e3bb466a70540dba24be89971523b78dbcf64a444eec4af9c6965065fddf04e6" exitCode=0 Aug 05 16:59:42.050296 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:42.050209 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-l2cfn" Aug 05 16:59:42.050296 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:42.050243 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-l2cfn" event={"ID":"3085d0d5-fb73-4f02-9dca-cdb3cf369470","Type":"ContainerDied","Data":"e3bb466a70540dba24be89971523b78dbcf64a444eec4af9c6965065fddf04e6"} Aug 05 16:59:42.050388 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:42.050298 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-l2cfn" event={"ID":"3085d0d5-fb73-4f02-9dca-cdb3cf369470","Type":"ContainerDied","Data":"c8aeb8b1602c2e44993e1ce32739f4b84049be0ebd9fa98284ad1dab4aedee16"} Aug 05 16:59:42.050388 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:42.050328 2573 scope.go:117] "RemoveContainer" containerID="e3bb466a70540dba24be89971523b78dbcf64a444eec4af9c6965065fddf04e6" Aug 05 16:59:42.059921 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:42.059905 2573 scope.go:117] "RemoveContainer" containerID="e3bb466a70540dba24be89971523b78dbcf64a444eec4af9c6965065fddf04e6" Aug 05 16:59:42.060159 ip-10-0-132-37 kubenswrapper[2573]: E0805 16:59:42.060140 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3bb466a70540dba24be89971523b78dbcf64a444eec4af9c6965065fddf04e6\": container with ID starting with e3bb466a70540dba24be89971523b78dbcf64a444eec4af9c6965065fddf04e6 not found: ID does not exist" containerID="e3bb466a70540dba24be89971523b78dbcf64a444eec4af9c6965065fddf04e6" Aug 05 16:59:42.060208 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:42.060168 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3bb466a70540dba24be89971523b78dbcf64a444eec4af9c6965065fddf04e6"} err="failed to get container status \"e3bb466a70540dba24be89971523b78dbcf64a444eec4af9c6965065fddf04e6\": rpc error: code = NotFound desc = could not find container \"e3bb466a70540dba24be89971523b78dbcf64a444eec4af9c6965065fddf04e6\": container with ID starting with e3bb466a70540dba24be89971523b78dbcf64a444eec4af9c6965065fddf04e6 not found: ID does not exist" Aug 05 16:59:42.073491 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:42.073469 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-l2cfn"] Aug 05 16:59:42.077206 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:42.077187 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-l2cfn"] Aug 05 16:59:42.142061 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:42.142013 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3085d0d5-fb73-4f02-9dca-cdb3cf369470-extensions-socket-volume\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:59:42.142061 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:42.142055 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkn7r\" (UniqueName: \"kubernetes.io/projected/3085d0d5-fb73-4f02-9dca-cdb3cf369470-kube-api-access-bkn7r\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 16:59:43.303125 ip-10-0-132-37 kubenswrapper[2573]: I0805 16:59:43.303090 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3085d0d5-fb73-4f02-9dca-cdb3cf369470" path="/var/lib/kubelet/pods/3085d0d5-fb73-4f02-9dca-cdb3cf369470/volumes" Aug 05 17:00:00.146237 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:00.146203 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29765820-6q747"] Aug 05 17:00:00.146649 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:00.146500 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="838cbb08-2f8f-4e7c-a74c-7d2e17928115" containerName="istio-proxy" Aug 05 17:00:00.146649 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:00.146510 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="838cbb08-2f8f-4e7c-a74c-7d2e17928115" containerName="istio-proxy" Aug 05 17:00:00.146649 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:00.146526 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3085d0d5-fb73-4f02-9dca-cdb3cf369470" containerName="manager" Aug 05 17:00:00.146649 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:00.146531 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3085d0d5-fb73-4f02-9dca-cdb3cf369470" containerName="manager" Aug 05 17:00:00.146649 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:00.146540 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="838cbb08-2f8f-4e7c-a74c-7d2e17928115" containerName="istio-proxy" Aug 05 17:00:00.146649 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:00.146545 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="838cbb08-2f8f-4e7c-a74c-7d2e17928115" containerName="istio-proxy" Aug 05 17:00:00.146649 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:00.146608 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="838cbb08-2f8f-4e7c-a74c-7d2e17928115" containerName="istio-proxy" Aug 05 17:00:00.146649 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:00.146616 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="838cbb08-2f8f-4e7c-a74c-7d2e17928115" containerName="istio-proxy" Aug 05 17:00:00.146649 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:00.146623 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3085d0d5-fb73-4f02-9dca-cdb3cf369470" containerName="manager" Aug 05 17:00:00.151031 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:00.151014 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765820-6q747" Aug 05 17:00:00.154593 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:00.154571 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q6h9g\"" Aug 05 17:00:00.158724 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:00.158699 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29765820-6q747"] Aug 05 17:00:00.161419 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:00.161401 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765820-6q747" Aug 05 17:00:00.290817 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:00.290716 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29765820-6q747"] Aug 05 17:00:00.293298 ip-10-0-132-37 kubenswrapper[2573]: W0805 17:00:00.293273 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03af45e_ed59_439d_9145_230a33c7df66.slice/crio-eed1a8ae54766b8d479ee79dfe779539e422c369d81c7bf9278a6617a5a02d2b WatchSource:0}: Error finding container eed1a8ae54766b8d479ee79dfe779539e422c369d81c7bf9278a6617a5a02d2b: Status 404 returned error can't find the container with id eed1a8ae54766b8d479ee79dfe779539e422c369d81c7bf9278a6617a5a02d2b Aug 05 17:00:01.136419 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:01.136357 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765820-6q747" event={"ID":"b03af45e-ed59-439d-9145-230a33c7df66","Type":"ContainerStarted","Data":"816482b99b6b7dca26e2f9446e422cc74dd7579a910a5c1634bee9ea5bca9d36"} Aug 05 17:00:01.136574 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:01.136437 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765820-6q747" event={"ID":"b03af45e-ed59-439d-9145-230a33c7df66","Type":"ContainerStarted","Data":"eed1a8ae54766b8d479ee79dfe779539e422c369d81c7bf9278a6617a5a02d2b"} Aug 05 17:00:01.154196 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:01.154148 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765820-6q747" podStartSLOduration=1.154132478 podStartE2EDuration="1.154132478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:00:01.153917648 +0000 UTC m=+2316.310763548" watchObservedRunningTime="2026-08-05 17:00:01.154132478 +0000 UTC m=+2316.310978393" Aug 05 17:00:02.142249 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:02.142210 2573 generic.go:356] "Generic (PLEG): container finished" podID="b03af45e-ed59-439d-9145-230a33c7df66" containerID="816482b99b6b7dca26e2f9446e422cc74dd7579a910a5c1634bee9ea5bca9d36" exitCode=0 Aug 05 17:00:02.142454 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:02.142299 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765820-6q747" event={"ID":"b03af45e-ed59-439d-9145-230a33c7df66","Type":"ContainerDied","Data":"816482b99b6b7dca26e2f9446e422cc74dd7579a910a5c1634bee9ea5bca9d36"} Aug 05 17:00:03.271561 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:03.271535 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765820-6q747" Aug 05 17:00:04.151587 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:04.151551 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765820-6q747" Aug 05 17:00:04.151587 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:04.151567 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765820-6q747" event={"ID":"b03af45e-ed59-439d-9145-230a33c7df66","Type":"ContainerDied","Data":"eed1a8ae54766b8d479ee79dfe779539e422c369d81c7bf9278a6617a5a02d2b"} Aug 05 17:00:04.151820 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:04.151635 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eed1a8ae54766b8d479ee79dfe779539e422c369d81c7bf9278a6617a5a02d2b" Aug 05 17:00:25.509158 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:25.509124 2573 scope.go:117] "RemoveContainer" containerID="81f6bb923c4eb7e1d0bcd1dbfbdca65dc98487c397fde6faaa68b83c59080622" Aug 05 17:00:25.518615 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:25.518595 2573 scope.go:117] "RemoveContainer" containerID="17e6ffda7bd4e030b589743f692738ec5bda10d4c8e5f270d3ebda4ff8903e51" Aug 05 17:00:54.128694 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:54.128650 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:00:57.277171 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:57.277128 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-thkmv"] Aug 05 17:00:57.277688 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:57.277571 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b03af45e-ed59-439d-9145-230a33c7df66" containerName="cleanup" Aug 05 17:00:57.277688 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:57.277588 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b03af45e-ed59-439d-9145-230a33c7df66" containerName="cleanup" Aug 05 17:00:57.277806 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:57.277690 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b03af45e-ed59-439d-9145-230a33c7df66" containerName="cleanup" Aug 05 17:00:57.280656 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:57.280635 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-thkmv" Aug 05 17:00:57.283926 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:57.283906 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vdhxn\"" Aug 05 17:00:57.295566 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:57.295539 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-thkmv"] Aug 05 17:00:57.362610 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:57.362572 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/962f33af-7194-4984-9aed-12c1bd3cb1ab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-thkmv\" (UID: \"962f33af-7194-4984-9aed-12c1bd3cb1ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-thkmv" Aug 05 17:00:57.362610 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:57.362622 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrxqg\" (UniqueName: \"kubernetes.io/projected/962f33af-7194-4984-9aed-12c1bd3cb1ab-kube-api-access-jrxqg\") pod \"kuadrant-operator-controller-manager-68cf589589-thkmv\" (UID: \"962f33af-7194-4984-9aed-12c1bd3cb1ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-thkmv" Aug 05 17:00:57.464052 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:57.464016 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrxqg\" (UniqueName: \"kubernetes.io/projected/962f33af-7194-4984-9aed-12c1bd3cb1ab-kube-api-access-jrxqg\") pod \"kuadrant-operator-controller-manager-68cf589589-thkmv\" (UID: \"962f33af-7194-4984-9aed-12c1bd3cb1ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-thkmv" Aug 05 17:00:57.464257 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:57.464109 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/962f33af-7194-4984-9aed-12c1bd3cb1ab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-thkmv\" (UID: \"962f33af-7194-4984-9aed-12c1bd3cb1ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-thkmv" Aug 05 17:00:57.464504 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:57.464487 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/962f33af-7194-4984-9aed-12c1bd3cb1ab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-thkmv\" (UID: \"962f33af-7194-4984-9aed-12c1bd3cb1ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-thkmv" Aug 05 17:00:57.474244 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:57.474218 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrxqg\" (UniqueName: \"kubernetes.io/projected/962f33af-7194-4984-9aed-12c1bd3cb1ab-kube-api-access-jrxqg\") pod \"kuadrant-operator-controller-manager-68cf589589-thkmv\" (UID: \"962f33af-7194-4984-9aed-12c1bd3cb1ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-thkmv" Aug 05 17:00:57.590630 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:57.590534 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-thkmv" Aug 05 17:00:57.730229 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:57.730105 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-thkmv"] Aug 05 17:00:57.734318 ip-10-0-132-37 kubenswrapper[2573]: W0805 17:00:57.733321 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod962f33af_7194_4984_9aed_12c1bd3cb1ab.slice/crio-bb2d3a4cb35969246ad8e8dd5776535b259965258317b07912f2e7f59172dde4 WatchSource:0}: Error finding container bb2d3a4cb35969246ad8e8dd5776535b259965258317b07912f2e7f59172dde4: Status 404 returned error can't find the container with id bb2d3a4cb35969246ad8e8dd5776535b259965258317b07912f2e7f59172dde4 Aug 05 17:00:58.382275 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:58.382231 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-thkmv" event={"ID":"962f33af-7194-4984-9aed-12c1bd3cb1ab","Type":"ContainerStarted","Data":"5884423f446d4c5d5ae440d0096719f6c56e6db0f2d628d8954325151f25c3a4"} Aug 05 17:00:58.382720 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:58.382323 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-thkmv" event={"ID":"962f33af-7194-4984-9aed-12c1bd3cb1ab","Type":"ContainerStarted","Data":"bb2d3a4cb35969246ad8e8dd5776535b259965258317b07912f2e7f59172dde4"} Aug 05 17:00:58.382720 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:58.382392 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-thkmv" Aug 05 17:00:58.408837 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:00:58.408777 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-thkmv" podStartSLOduration=1.408758824 podStartE2EDuration="1.408758824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:00:58.408296995 +0000 UTC m=+2373.565142898" watchObservedRunningTime="2026-08-05 17:00:58.408758824 +0000 UTC m=+2373.565604728" Aug 05 17:01:09.388509 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:09.388475 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-thkmv" Aug 05 17:01:15.907120 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:15.907074 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:01:16.250325 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.250241 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h"] Aug 05 17:01:16.253645 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.253629 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.257781 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.257761 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d2645e95-openshift-default-dockercfg-29xcs\"" Aug 05 17:01:16.274108 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.274080 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h"] Aug 05 17:01:16.324458 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.324404 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-workload-socket\") pod \"e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.324458 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.324444 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istio-podinfo\") pod \"e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.324458 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.324471 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istio-token\") pod \"e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.324895 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.324539 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-workload-certs\") pod \"e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.324895 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.324579 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-credential-socket\") pod \"e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.324895 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.324612 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istio-envoy\") pod \"e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.324895 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.324633 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29xfs\" (UniqueName: \"kubernetes.io/projected/abe1b190-54ab-4de3-ac85-bb73890d7d2c-kube-api-access-29xfs\") pod \"e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.324895 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.324708 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istio-data\") pod \"e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.324895 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.324730 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istiod-ca-cert\") pod \"e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.425472 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.425429 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-workload-socket\") pod \"e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.425472 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.425475 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istio-podinfo\") pod \"e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.425736 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.425501 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istio-token\") pod \"e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.425736 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.425548 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-workload-certs\") pod \"e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.425736 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.425575 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-credential-socket\") pod \"e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.425736 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.425607 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istio-envoy\") pod \"e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.425736 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.425628 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29xfs\" (UniqueName: \"kubernetes.io/projected/abe1b190-54ab-4de3-ac85-bb73890d7d2c-kube-api-access-29xfs\") pod \"e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.425736 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.425717 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istio-data\") pod \"e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.426019 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.425741 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istiod-ca-cert\") pod \"e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.426081 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.426038 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-workload-socket\") pod \"e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.426081 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.426068 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-credential-socket\") pod \"e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.426197 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.426132 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-workload-certs\") pod \"e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.426247 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.426234 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istio-data\") pod \"e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.426507 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.426486 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istiod-ca-cert\") pod \"e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.428048 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.428003 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istio-envoy\") pod \"e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.428512 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.428491 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istio-podinfo\") pod \"e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.436591 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.436565 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istio-token\") pod \"e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.436714 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.436682 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29xfs\" (UniqueName: \"kubernetes.io/projected/abe1b190-54ab-4de3-ac85-bb73890d7d2c-kube-api-access-29xfs\") pod \"e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.564222 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.564193 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:16.596954 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.596919 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:01:16.704391 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.704350 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h"] Aug 05 17:01:16.706814 ip-10-0-132-37 kubenswrapper[2573]: W0805 17:01:16.706787 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe1b190_54ab_4de3_ac85_bb73890d7d2c.slice/crio-aba14ec40f21ebadceeba5a05d273222ccd792fd20d3715dced153c6cdcd5a16 WatchSource:0}: Error finding container aba14ec40f21ebadceeba5a05d273222ccd792fd20d3715dced153c6cdcd5a16: Status 404 returned error can't find the container with id aba14ec40f21ebadceeba5a05d273222ccd792fd20d3715dced153c6cdcd5a16 Aug 05 17:01:16.709100 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.709050 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:01:16.709189 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.709147 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:01:16.709247 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:16.709190 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:01:17.306029 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:17.305981 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:01:17.465368 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:17.465331 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" event={"ID":"abe1b190-54ab-4de3-ac85-bb73890d7d2c","Type":"ContainerStarted","Data":"0bf74d7e92fd6cb9125943fac4a3be0cb9706dc9e6c9288fe650b290486dbd78"} Aug 05 17:01:17.465576 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:17.465401 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" event={"ID":"abe1b190-54ab-4de3-ac85-bb73890d7d2c","Type":"ContainerStarted","Data":"aba14ec40f21ebadceeba5a05d273222ccd792fd20d3715dced153c6cdcd5a16"} Aug 05 17:01:17.491915 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:17.491858 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" podStartSLOduration=1.491842203 podStartE2EDuration="1.491842203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:01:17.488561997 +0000 UTC m=+2392.645407898" watchObservedRunningTime="2026-08-05 17:01:17.491842203 +0000 UTC m=+2392.648688103" Aug 05 17:01:17.565068 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:17.564977 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:17.569825 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:17.569802 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:18.470129 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:18.470098 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:18.471181 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:18.471163 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:19.520250 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:19.520216 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:01:19.622370 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:19.622334 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:01:20.602128 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:20.602098 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:01:20.914108 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:20.914023 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:01:21.276273 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:21.276234 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:01:26.203288 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:26.203250 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:01:33.677744 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:33.677335 2573 patch_prober.go:28] interesting pod/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": read tcp 10.133.0.2:34912->10.133.0.48:15021: read: connection reset by peer" start-of-body= Aug 05 17:01:33.677744 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:33.677430 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" podUID="abe1b190-54ab-4de3-ac85-bb73890d7d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": read tcp 10.133.0.2:34912->10.133.0.48:15021: read: connection reset by peer" Aug 05 17:01:34.542321 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:34.542290 2573 generic.go:356] "Generic (PLEG): container finished" podID="abe1b190-54ab-4de3-ac85-bb73890d7d2c" containerID="0bf74d7e92fd6cb9125943fac4a3be0cb9706dc9e6c9288fe650b290486dbd78" exitCode=0 Aug 05 17:01:34.542506 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:34.542363 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" event={"ID":"abe1b190-54ab-4de3-ac85-bb73890d7d2c","Type":"ContainerDied","Data":"0bf74d7e92fd6cb9125943fac4a3be0cb9706dc9e6c9288fe650b290486dbd78"} Aug 05 17:01:34.542789 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:34.542771 2573 scope.go:117] "RemoveContainer" containerID="0bf74d7e92fd6cb9125943fac4a3be0cb9706dc9e6c9288fe650b290486dbd78" Aug 05 17:01:34.543838 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:34.543805 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:01:34.543912 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:34.543895 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:01:34.543964 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:34.543948 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:01:35.552105 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:35.552067 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" event={"ID":"abe1b190-54ab-4de3-ac85-bb73890d7d2c","Type":"ContainerStarted","Data":"d3507b2160cc2e6921669e79a1cad412a72f28299ca59b90d6ef5bd8caa831b6"} Aug 05 17:01:35.564966 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:35.564945 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:35.565141 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:35.565120 2573 patch_prober.go:28] interesting pod/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Aug 05 17:01:35.565190 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:35.565163 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" podUID="abe1b190-54ab-4de3-ac85-bb73890d7d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Aug 05 17:01:36.565026 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:36.564990 2573 patch_prober.go:28] interesting pod/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Aug 05 17:01:36.565434 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:36.565041 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" podUID="abe1b190-54ab-4de3-ac85-bb73890d7d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Aug 05 17:01:37.565002 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:37.564968 2573 patch_prober.go:28] interesting pod/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Aug 05 17:01:37.565179 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:37.565032 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" podUID="abe1b190-54ab-4de3-ac85-bb73890d7d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Aug 05 17:01:38.564801 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:38.564766 2573 patch_prober.go:28] interesting pod/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Aug 05 17:01:38.564975 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:38.564824 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" podUID="abe1b190-54ab-4de3-ac85-bb73890d7d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Aug 05 17:01:39.565053 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:39.565018 2573 patch_prober.go:28] interesting pod/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Aug 05 17:01:39.565594 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:39.565567 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" podUID="abe1b190-54ab-4de3-ac85-bb73890d7d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Aug 05 17:01:40.564645 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:40.564604 2573 patch_prober.go:28] interesting pod/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Aug 05 17:01:40.564834 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:40.564664 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" podUID="abe1b190-54ab-4de3-ac85-bb73890d7d2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Aug 05 17:01:41.568648 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:41.568611 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:41.569022 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:41.568901 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:41.569713 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:41.569696 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:01:44.427237 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:44.427205 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:01:50.917507 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:50.917412 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2"] Aug 05 17:01:50.920050 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:50.920028 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" Aug 05 17:01:50.923604 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:50.923582 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d2645e95\"/\"kube-root-ca.crt\"" Aug 05 17:01:50.925275 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:50.925247 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d2645e95\"/\"e2e-lifecycle-model-d2645e95-kserve-self-signed-certs\"" Aug 05 17:01:50.925385 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:50.925306 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d2645e95\"/\"openshift-service-ca.crt\"" Aug 05 17:01:50.925385 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:50.925318 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d2645e95\"/\"default-dockercfg-x6g96\"" Aug 05 17:01:50.934141 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:50.934119 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2"] Aug 05 17:01:51.000283 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:51.000250 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:01:51.040510 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:51.040478 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d2fc6e0-9f13-48bc-9309-83018f691ca3-model-cache\") pod \"e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2\" (UID: \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\") " pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" Aug 05 17:01:51.040670 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:51.040517 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d2fc6e0-9f13-48bc-9309-83018f691ca3-dshm\") pod \"e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2\" (UID: \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\") " pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" Aug 05 17:01:51.040670 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:51.040543 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d2fc6e0-9f13-48bc-9309-83018f691ca3-tls-certs\") pod \"e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2\" (UID: \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\") " pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" Aug 05 17:01:51.040670 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:51.040632 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d2fc6e0-9f13-48bc-9309-83018f691ca3-home\") pod \"e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2\" (UID: \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\") " pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" Aug 05 17:01:51.040670 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:51.040649 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpfbq\" (UniqueName: \"kubernetes.io/projected/3d2fc6e0-9f13-48bc-9309-83018f691ca3-kube-api-access-kpfbq\") pod \"e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2\" (UID: \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\") " pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" Aug 05 17:01:51.040670 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:51.040668 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d2fc6e0-9f13-48bc-9309-83018f691ca3-tmp-dir\") pod \"e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2\" (UID: \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\") " pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" Aug 05 17:01:51.142030 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:51.141996 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d2fc6e0-9f13-48bc-9309-83018f691ca3-home\") pod \"e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2\" (UID: \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\") " pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" Aug 05 17:01:51.142030 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:51.142030 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpfbq\" (UniqueName: \"kubernetes.io/projected/3d2fc6e0-9f13-48bc-9309-83018f691ca3-kube-api-access-kpfbq\") pod \"e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2\" (UID: \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\") " pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" Aug 05 17:01:51.142339 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:51.142053 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d2fc6e0-9f13-48bc-9309-83018f691ca3-tmp-dir\") pod \"e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2\" (UID: \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\") " pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" Aug 05 17:01:51.142339 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:51.142101 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d2fc6e0-9f13-48bc-9309-83018f691ca3-model-cache\") pod \"e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2\" (UID: \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\") " pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" Aug 05 17:01:51.142339 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:51.142128 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d2fc6e0-9f13-48bc-9309-83018f691ca3-dshm\") pod \"e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2\" (UID: \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\") " pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" Aug 05 17:01:51.142339 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:51.142156 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d2fc6e0-9f13-48bc-9309-83018f691ca3-tls-certs\") pod \"e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2\" (UID: \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\") " pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" Aug 05 17:01:51.142588 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:51.142475 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d2fc6e0-9f13-48bc-9309-83018f691ca3-home\") pod \"e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2\" (UID: \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\") " pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" Aug 05 17:01:51.142659 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:51.142636 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d2fc6e0-9f13-48bc-9309-83018f691ca3-model-cache\") pod \"e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2\" (UID: \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\") " pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" Aug 05 17:01:51.142980 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:51.142953 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d2fc6e0-9f13-48bc-9309-83018f691ca3-tmp-dir\") pod \"e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2\" (UID: \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\") " pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" Aug 05 17:01:51.145048 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:51.145022 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d2fc6e0-9f13-48bc-9309-83018f691ca3-dshm\") pod \"e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2\" (UID: \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\") " pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" Aug 05 17:01:51.145284 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:51.145259 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d2fc6e0-9f13-48bc-9309-83018f691ca3-tls-certs\") pod \"e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2\" (UID: \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\") " pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" Aug 05 17:01:51.151690 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:51.151654 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpfbq\" (UniqueName: \"kubernetes.io/projected/3d2fc6e0-9f13-48bc-9309-83018f691ca3-kube-api-access-kpfbq\") pod \"e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2\" (UID: \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\") " pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" Aug 05 17:01:51.229408 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:51.229283 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" Aug 05 17:01:51.364927 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:51.364898 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2"] Aug 05 17:01:51.367084 ip-10-0-132-37 kubenswrapper[2573]: W0805 17:01:51.367054 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d2fc6e0_9f13_48bc_9309_83018f691ca3.slice/crio-bc1ddf69c4ae3b3737737f8da825fe9860b91fafd758f493981678c12ed4066b WatchSource:0}: Error finding container bc1ddf69c4ae3b3737737f8da825fe9860b91fafd758f493981678c12ed4066b: Status 404 returned error can't find the container with id bc1ddf69c4ae3b3737737f8da825fe9860b91fafd758f493981678c12ed4066b Aug 05 17:01:51.624730 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:51.624693 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" event={"ID":"3d2fc6e0-9f13-48bc-9309-83018f691ca3","Type":"ContainerStarted","Data":"016790740a48cf0de24bc5290843f6b9669bc37307863879d5578d917daf4f44"} Aug 05 17:01:51.624909 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:51.624738 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" event={"ID":"3d2fc6e0-9f13-48bc-9309-83018f691ca3","Type":"ContainerStarted","Data":"bc1ddf69c4ae3b3737737f8da825fe9860b91fafd758f493981678c12ed4066b"} Aug 05 17:01:51.646317 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:01:51.646261 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" podStartSLOduration=1.646244995 podStartE2EDuration="1.646244995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:01:51.645606145 +0000 UTC m=+2426.802452047" watchObservedRunningTime="2026-08-05 17:01:51.646244995 +0000 UTC m=+2426.803090895" Aug 05 17:02:01.230070 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:01.230025 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" Aug 05 17:02:01.230537 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:01.230084 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" Aug 05 17:02:01.242786 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:01.242756 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" Aug 05 17:02:01.680880 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:01.680847 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" Aug 05 17:02:23.172260 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:23.172222 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:02:25.552214 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:25.552185 2573 scope.go:117] "RemoveContainer" containerID="f9a6b9f4884f03d5f89a7e5cf8c912df810c1a218c9e2687803261aae870b18d" Aug 05 17:02:34.710837 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:34.710800 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:02:34.826787 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:34.826753 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:02:34.942156 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:34.942123 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:02:35.053608 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:35.053573 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:02:38.105402 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:38.105351 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:02:43.039912 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:43.039875 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2"] Aug 05 17:02:43.139241 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:43.139207 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:02:58.058650 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.058584 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" podUID="3d2fc6e0-9f13-48bc-9309-83018f691ca3" containerName="main" containerID="cri-o://016790740a48cf0de24bc5290843f6b9669bc37307863879d5578d917daf4f44" gracePeriod=45 Aug 05 17:02:58.298733 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.298707 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" Aug 05 17:02:58.344754 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.344681 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d2fc6e0-9f13-48bc-9309-83018f691ca3-model-cache\") pod \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\" (UID: \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\") " Aug 05 17:02:58.344754 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.344723 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpfbq\" (UniqueName: \"kubernetes.io/projected/3d2fc6e0-9f13-48bc-9309-83018f691ca3-kube-api-access-kpfbq\") pod \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\" (UID: \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\") " Aug 05 17:02:58.344954 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.344760 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d2fc6e0-9f13-48bc-9309-83018f691ca3-tmp-dir\") pod \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\" (UID: \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\") " Aug 05 17:02:58.344954 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.344888 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d2fc6e0-9f13-48bc-9309-83018f691ca3-home\") pod \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\" (UID: \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\") " Aug 05 17:02:58.345070 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.345013 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d2fc6e0-9f13-48bc-9309-83018f691ca3-dshm\") pod \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\" (UID: \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\") " Aug 05 17:02:58.345070 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.345010 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2fc6e0-9f13-48bc-9309-83018f691ca3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d2fc6e0-9f13-48bc-9309-83018f691ca3" (UID: "3d2fc6e0-9f13-48bc-9309-83018f691ca3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:02:58.345070 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.345046 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d2fc6e0-9f13-48bc-9309-83018f691ca3-tls-certs\") pod \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\" (UID: \"3d2fc6e0-9f13-48bc-9309-83018f691ca3\") " Aug 05 17:02:58.345070 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.345026 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2fc6e0-9f13-48bc-9309-83018f691ca3-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d2fc6e0-9f13-48bc-9309-83018f691ca3" (UID: "3d2fc6e0-9f13-48bc-9309-83018f691ca3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:02:58.345276 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.345157 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2fc6e0-9f13-48bc-9309-83018f691ca3-home" (OuterVolumeSpecName: "home") pod "3d2fc6e0-9f13-48bc-9309-83018f691ca3" (UID: "3d2fc6e0-9f13-48bc-9309-83018f691ca3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:02:58.345400 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.345349 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d2fc6e0-9f13-48bc-9309-83018f691ca3-model-cache\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:02:58.345400 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.345391 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d2fc6e0-9f13-48bc-9309-83018f691ca3-tmp-dir\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:02:58.345549 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.345405 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d2fc6e0-9f13-48bc-9309-83018f691ca3-home\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:02:58.347577 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.347546 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2fc6e0-9f13-48bc-9309-83018f691ca3-dshm" (OuterVolumeSpecName: "dshm") pod "3d2fc6e0-9f13-48bc-9309-83018f691ca3" (UID: "3d2fc6e0-9f13-48bc-9309-83018f691ca3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:02:58.347691 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.347625 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2fc6e0-9f13-48bc-9309-83018f691ca3-kube-api-access-kpfbq" (OuterVolumeSpecName: "kube-api-access-kpfbq") pod "3d2fc6e0-9f13-48bc-9309-83018f691ca3" (UID: "3d2fc6e0-9f13-48bc-9309-83018f691ca3"). InnerVolumeSpecName "kube-api-access-kpfbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:02:58.347691 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.347636 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d2fc6e0-9f13-48bc-9309-83018f691ca3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d2fc6e0-9f13-48bc-9309-83018f691ca3" (UID: "3d2fc6e0-9f13-48bc-9309-83018f691ca3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:02:58.445812 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.445776 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d2fc6e0-9f13-48bc-9309-83018f691ca3-dshm\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:02:58.445812 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.445806 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d2fc6e0-9f13-48bc-9309-83018f691ca3-tls-certs\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:02:58.445812 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.445817 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpfbq\" (UniqueName: \"kubernetes.io/projected/3d2fc6e0-9f13-48bc-9309-83018f691ca3-kube-api-access-kpfbq\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:02:58.936279 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.936240 2573 generic.go:356] "Generic (PLEG): container finished" podID="3d2fc6e0-9f13-48bc-9309-83018f691ca3" containerID="016790740a48cf0de24bc5290843f6b9669bc37307863879d5578d917daf4f44" exitCode=0 Aug 05 17:02:58.936502 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.936292 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" Aug 05 17:02:58.936502 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.936306 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" event={"ID":"3d2fc6e0-9f13-48bc-9309-83018f691ca3","Type":"ContainerDied","Data":"016790740a48cf0de24bc5290843f6b9669bc37307863879d5578d917daf4f44"} Aug 05 17:02:58.936502 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.936345 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2" event={"ID":"3d2fc6e0-9f13-48bc-9309-83018f691ca3","Type":"ContainerDied","Data":"bc1ddf69c4ae3b3737737f8da825fe9860b91fafd758f493981678c12ed4066b"} Aug 05 17:02:58.936502 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.936366 2573 scope.go:117] "RemoveContainer" containerID="016790740a48cf0de24bc5290843f6b9669bc37307863879d5578d917daf4f44" Aug 05 17:02:58.945360 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.945339 2573 scope.go:117] "RemoveContainer" containerID="016790740a48cf0de24bc5290843f6b9669bc37307863879d5578d917daf4f44" Aug 05 17:02:58.945621 ip-10-0-132-37 kubenswrapper[2573]: E0805 17:02:58.945601 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"016790740a48cf0de24bc5290843f6b9669bc37307863879d5578d917daf4f44\": container with ID starting with 016790740a48cf0de24bc5290843f6b9669bc37307863879d5578d917daf4f44 not found: ID does not exist" containerID="016790740a48cf0de24bc5290843f6b9669bc37307863879d5578d917daf4f44" Aug 05 17:02:58.945684 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.945629 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"016790740a48cf0de24bc5290843f6b9669bc37307863879d5578d917daf4f44"} err="failed to get container status \"016790740a48cf0de24bc5290843f6b9669bc37307863879d5578d917daf4f44\": rpc error: code = NotFound desc = could not find container \"016790740a48cf0de24bc5290843f6b9669bc37307863879d5578d917daf4f44\": container with ID starting with 016790740a48cf0de24bc5290843f6b9669bc37307863879d5578d917daf4f44 not found: ID does not exist" Aug 05 17:02:58.967639 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.967613 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2"] Aug 05 17:02:58.977881 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:58.977859 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-d2645e95/e2e-lifecycle-model-d2645e95-kserve-665b8b9499-pwsw2"] Aug 05 17:02:59.304293 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:02:59.304259 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d2fc6e0-9f13-48bc-9309-83018f691ca3" path="/var/lib/kubelet/pods/3d2fc6e0-9f13-48bc-9309-83018f691ca3/volumes" Aug 05 17:03:19.550766 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:19.550724 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:03:20.822801 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:20.822768 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:03:26.731884 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:26.731848 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:03:26.831808 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:26.831778 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:03:27.013981 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:27.013940 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h"] Aug 05 17:03:27.014226 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:27.014201 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" podUID="abe1b190-54ab-4de3-ac85-bb73890d7d2c" containerName="istio-proxy" containerID="cri-o://d3507b2160cc2e6921669e79a1cad412a72f28299ca59b90d6ef5bd8caa831b6" gracePeriod=30 Aug 05 17:03:27.047894 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:27.047858 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:03:27.272217 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:27.272125 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:03:32.253078 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:32.253051 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:03:32.346269 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:32.346182 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29xfs\" (UniqueName: \"kubernetes.io/projected/abe1b190-54ab-4de3-ac85-bb73890d7d2c-kube-api-access-29xfs\") pod \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " Aug 05 17:03:32.346269 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:32.346245 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istio-podinfo\") pod \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " Aug 05 17:03:32.346269 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:32.346262 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istiod-ca-cert\") pod \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " Aug 05 17:03:32.346524 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:32.346281 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istio-data\") pod \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " Aug 05 17:03:32.346524 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:32.346456 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-credential-socket\") pod \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " Aug 05 17:03:32.346524 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:32.346505 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-workload-socket\") pod \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " Aug 05 17:03:32.346662 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:32.346535 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istio-envoy\") pod \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " Aug 05 17:03:32.346662 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:32.346558 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istio-token\") pod \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " Aug 05 17:03:32.346662 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:32.346576 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istio-data" (OuterVolumeSpecName: "istio-data") pod "abe1b190-54ab-4de3-ac85-bb73890d7d2c" (UID: "abe1b190-54ab-4de3-ac85-bb73890d7d2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:03:32.346662 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:32.346588 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-workload-certs\") pod \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\" (UID: \"abe1b190-54ab-4de3-ac85-bb73890d7d2c\") " Aug 05 17:03:32.346874 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:32.346703 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "abe1b190-54ab-4de3-ac85-bb73890d7d2c" (UID: "abe1b190-54ab-4de3-ac85-bb73890d7d2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:03:32.346874 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:32.346728 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "abe1b190-54ab-4de3-ac85-bb73890d7d2c" (UID: "abe1b190-54ab-4de3-ac85-bb73890d7d2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:03:32.346874 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:32.346792 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "abe1b190-54ab-4de3-ac85-bb73890d7d2c" (UID: "abe1b190-54ab-4de3-ac85-bb73890d7d2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:03:32.347103 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:32.346927 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istio-data\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:03:32.347103 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:32.346948 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-credential-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:03:32.347103 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:32.346964 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-workload-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:03:32.347103 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:32.346980 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istiod-ca-cert\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:03:32.347103 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:32.346993 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "abe1b190-54ab-4de3-ac85-bb73890d7d2c" (UID: "abe1b190-54ab-4de3-ac85-bb73890d7d2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:03:32.348558 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:32.348534 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe1b190-54ab-4de3-ac85-bb73890d7d2c-kube-api-access-29xfs" (OuterVolumeSpecName: "kube-api-access-29xfs") pod "abe1b190-54ab-4de3-ac85-bb73890d7d2c" (UID: "abe1b190-54ab-4de3-ac85-bb73890d7d2c"). InnerVolumeSpecName "kube-api-access-29xfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:03:32.348661 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:32.348562 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "abe1b190-54ab-4de3-ac85-bb73890d7d2c" (UID: "abe1b190-54ab-4de3-ac85-bb73890d7d2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 17:03:32.348661 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:32.348639 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istio-token" (OuterVolumeSpecName: "istio-token") pod "abe1b190-54ab-4de3-ac85-bb73890d7d2c" (UID: "abe1b190-54ab-4de3-ac85-bb73890d7d2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:03:32.348771 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:32.348734 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "abe1b190-54ab-4de3-ac85-bb73890d7d2c" (UID: "abe1b190-54ab-4de3-ac85-bb73890d7d2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:03:32.447489 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:32.447456 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istio-podinfo\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:03:32.447489 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:32.447481 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istio-envoy\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:03:32.447489 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:32.447490 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe1b190-54ab-4de3-ac85-bb73890d7d2c-istio-token\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:03:32.447489 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:32.447497 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe1b190-54ab-4de3-ac85-bb73890d7d2c-workload-certs\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:03:32.447734 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:32.447507 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29xfs\" (UniqueName: \"kubernetes.io/projected/abe1b190-54ab-4de3-ac85-bb73890d7d2c-kube-api-access-29xfs\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:03:33.096335 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:33.096302 2573 generic.go:356] "Generic (PLEG): container finished" podID="abe1b190-54ab-4de3-ac85-bb73890d7d2c" containerID="d3507b2160cc2e6921669e79a1cad412a72f28299ca59b90d6ef5bd8caa831b6" exitCode=0 Aug 05 17:03:33.096564 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:33.096342 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" event={"ID":"abe1b190-54ab-4de3-ac85-bb73890d7d2c","Type":"ContainerDied","Data":"d3507b2160cc2e6921669e79a1cad412a72f28299ca59b90d6ef5bd8caa831b6"} Aug 05 17:03:33.099960 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:33.096733 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" event={"ID":"abe1b190-54ab-4de3-ac85-bb73890d7d2c","Type":"ContainerDied","Data":"aba14ec40f21ebadceeba5a05d273222ccd792fd20d3715dced153c6cdcd5a16"} Aug 05 17:03:33.099960 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:33.096821 2573 scope.go:117] "RemoveContainer" containerID="d3507b2160cc2e6921669e79a1cad412a72f28299ca59b90d6ef5bd8caa831b6" Aug 05 17:03:33.099960 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:33.096885 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h" Aug 05 17:03:33.110677 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:33.110660 2573 scope.go:117] "RemoveContainer" containerID="0bf74d7e92fd6cb9125943fac4a3be0cb9706dc9e6c9288fe650b290486dbd78" Aug 05 17:03:33.118046 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:33.118028 2573 scope.go:117] "RemoveContainer" containerID="d3507b2160cc2e6921669e79a1cad412a72f28299ca59b90d6ef5bd8caa831b6" Aug 05 17:03:33.118276 ip-10-0-132-37 kubenswrapper[2573]: E0805 17:03:33.118256 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3507b2160cc2e6921669e79a1cad412a72f28299ca59b90d6ef5bd8caa831b6\": container with ID starting with d3507b2160cc2e6921669e79a1cad412a72f28299ca59b90d6ef5bd8caa831b6 not found: ID does not exist" containerID="d3507b2160cc2e6921669e79a1cad412a72f28299ca59b90d6ef5bd8caa831b6" Aug 05 17:03:33.118322 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:33.118283 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3507b2160cc2e6921669e79a1cad412a72f28299ca59b90d6ef5bd8caa831b6"} err="failed to get container status \"d3507b2160cc2e6921669e79a1cad412a72f28299ca59b90d6ef5bd8caa831b6\": rpc error: code = NotFound desc = could not find container \"d3507b2160cc2e6921669e79a1cad412a72f28299ca59b90d6ef5bd8caa831b6\": container with ID starting with d3507b2160cc2e6921669e79a1cad412a72f28299ca59b90d6ef5bd8caa831b6 not found: ID does not exist" Aug 05 17:03:33.118322 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:33.118298 2573 scope.go:117] "RemoveContainer" containerID="0bf74d7e92fd6cb9125943fac4a3be0cb9706dc9e6c9288fe650b290486dbd78" Aug 05 17:03:33.118558 ip-10-0-132-37 kubenswrapper[2573]: E0805 17:03:33.118541 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bf74d7e92fd6cb9125943fac4a3be0cb9706dc9e6c9288fe650b290486dbd78\": container with ID starting with 0bf74d7e92fd6cb9125943fac4a3be0cb9706dc9e6c9288fe650b290486dbd78 not found: ID does not exist" containerID="0bf74d7e92fd6cb9125943fac4a3be0cb9706dc9e6c9288fe650b290486dbd78" Aug 05 17:03:33.118601 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:33.118563 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bf74d7e92fd6cb9125943fac4a3be0cb9706dc9e6c9288fe650b290486dbd78"} err="failed to get container status \"0bf74d7e92fd6cb9125943fac4a3be0cb9706dc9e6c9288fe650b290486dbd78\": rpc error: code = NotFound desc = could not find container \"0bf74d7e92fd6cb9125943fac4a3be0cb9706dc9e6c9288fe650b290486dbd78\": container with ID starting with 0bf74d7e92fd6cb9125943fac4a3be0cb9706dc9e6c9288fe650b290486dbd78 not found: ID does not exist" Aug 05 17:03:33.130230 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:33.130206 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h"] Aug 05 17:03:33.138739 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:33.138717 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d2645e95-openshift-default-65684b4d56-8q58h"] Aug 05 17:03:33.303988 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:33.303955 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abe1b190-54ab-4de3-ac85-bb73890d7d2c" path="/var/lib/kubelet/pods/abe1b190-54ab-4de3-ac85-bb73890d7d2c/volumes" Aug 05 17:03:38.777213 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:38.777172 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:03:39.072285 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:39.072195 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:03:39.681989 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:39.681949 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:03:39.781276 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:39.781239 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:03:39.919062 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:39.919026 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:03:41.188415 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:41.188362 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:03:47.863706 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:47.863669 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:03:47.941568 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:47.941534 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:03:48.534275 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:48.534234 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:03:55.289012 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:55.288978 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:03:55.389102 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:55.389067 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:03:55.489096 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:03:55.489060 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:05.960763 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:05.960728 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:06.166327 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:06.166268 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:27.129078 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:27.129042 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:33.378990 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:33.378953 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:33.453340 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:33.453304 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:33.788183 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:33.788154 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:34.050520 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.050439 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc"] Aug 05 17:04:34.050766 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.050751 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe1b190-54ab-4de3-ac85-bb73890d7d2c" containerName="istio-proxy" Aug 05 17:04:34.050819 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.050767 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe1b190-54ab-4de3-ac85-bb73890d7d2c" containerName="istio-proxy" Aug 05 17:04:34.050819 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.050791 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d2fc6e0-9f13-48bc-9309-83018f691ca3" containerName="main" Aug 05 17:04:34.050819 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.050796 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d2fc6e0-9f13-48bc-9309-83018f691ca3" containerName="main" Aug 05 17:04:34.050923 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.050842 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe1b190-54ab-4de3-ac85-bb73890d7d2c" containerName="istio-proxy" Aug 05 17:04:34.050923 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.050850 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d2fc6e0-9f13-48bc-9309-83018f691ca3" containerName="main" Aug 05 17:04:34.053668 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.053651 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.057729 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.057705 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-09f3f8-openshift-default-dockercfg-s9cz6\"" Aug 05 17:04:34.069701 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.069666 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc"] Aug 05 17:04:34.081290 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.081263 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:34.195182 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.195145 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-workload-certs\") pod \"e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.195426 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.195193 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istio-podinfo\") pod \"e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.195426 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.195213 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istio-token\") pod \"e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.195426 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.195271 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istio-envoy\") pod \"e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.195426 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.195328 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8bvr\" (UniqueName: \"kubernetes.io/projected/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-kube-api-access-n8bvr\") pod \"e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.195426 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.195409 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istiod-ca-cert\") pod \"e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.195712 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.195435 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-workload-socket\") pod \"e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.195712 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.195468 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-credential-socket\") pod \"e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.195712 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.195533 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istio-data\") pod \"e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.296720 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.296679 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istiod-ca-cert\") pod \"e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.296720 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.296725 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-workload-socket\") pod \"e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.296988 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.296750 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-credential-socket\") pod \"e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.296988 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.296791 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istio-data\") pod \"e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.296988 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.296819 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-workload-certs\") pod \"e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.296988 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.296859 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istio-podinfo\") pod \"e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.296988 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.296884 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istio-token\") pod \"e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.296988 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.296914 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istio-envoy\") pod \"e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.296988 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.296936 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8bvr\" (UniqueName: \"kubernetes.io/projected/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-kube-api-access-n8bvr\") pod \"e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.297337 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.297185 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-workload-socket\") pod \"e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.297337 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.297227 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istio-data\") pod \"e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.297337 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.297254 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-credential-socket\") pod \"e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.297549 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.297468 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istiod-ca-cert\") pod \"e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.297549 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.297504 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-workload-certs\") pod \"e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.299728 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.299698 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istio-envoy\") pod \"e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.299853 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.299839 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istio-podinfo\") pod \"e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.306073 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.306004 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istio-token\") pod \"e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.306357 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.306326 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8bvr\" (UniqueName: \"kubernetes.io/projected/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-kube-api-access-n8bvr\") pod \"e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.364237 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.364205 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:34.384644 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.384616 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:34.504977 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.504952 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc"] Aug 05 17:04:34.507546 ip-10-0-132-37 kubenswrapper[2573]: W0805 17:04:34.507513 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3d2cac_d8c1_40e3_86b1_ae274f842e03.slice/crio-72035fbde5023b8d30a485ecc0773edf927ef1d47f47c525e503c24b624e3edb WatchSource:0}: Error finding container 72035fbde5023b8d30a485ecc0773edf927ef1d47f47c525e503c24b624e3edb: Status 404 returned error can't find the container with id 72035fbde5023b8d30a485ecc0773edf927ef1d47f47c525e503c24b624e3edb Aug 05 17:04:34.509879 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.509843 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:04:34.509947 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.509931 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:04:34.509986 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:34.509971 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:04:35.369273 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:35.369239 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" event={"ID":"1c3d2cac-d8c1-40e3-86b1-ae274f842e03","Type":"ContainerStarted","Data":"f34f7fcdbcf90f16c38e81d0c65d9e5c27c51d1928880fa87711454971b96924"} Aug 05 17:04:35.369483 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:35.369291 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" event={"ID":"1c3d2cac-d8c1-40e3-86b1-ae274f842e03","Type":"ContainerStarted","Data":"72035fbde5023b8d30a485ecc0773edf927ef1d47f47c525e503c24b624e3edb"} Aug 05 17:04:35.400287 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:35.400228 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" podStartSLOduration=1.400212308 podStartE2EDuration="1.400212308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:04:35.397740262 +0000 UTC m=+2590.554586167" watchObservedRunningTime="2026-08-05 17:04:35.400212308 +0000 UTC m=+2590.557058209" Aug 05 17:04:36.364369 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:36.364334 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:36.368924 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:36.368900 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:36.373768 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:36.373739 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:36.374441 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:36.374424 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:37.417629 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:37.417593 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:37.522482 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:37.522448 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:38.549761 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:38.549727 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:39.144758 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:39.144723 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:39.257505 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:39.257472 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:04:51.610660 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:51.610135 2573 patch_prober.go:28] interesting pod/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": read tcp 10.133.0.2:45394->10.133.0.50:15021: read: connection reset by peer" start-of-body= Aug 05 17:04:51.610660 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:51.610216 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" podUID="1c3d2cac-d8c1-40e3-86b1-ae274f842e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": read tcp 10.133.0.2:45394->10.133.0.50:15021: read: connection reset by peer" Aug 05 17:04:52.441660 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:52.441625 2573 generic.go:356] "Generic (PLEG): container finished" podID="1c3d2cac-d8c1-40e3-86b1-ae274f842e03" containerID="f34f7fcdbcf90f16c38e81d0c65d9e5c27c51d1928880fa87711454971b96924" exitCode=0 Aug 05 17:04:52.441660 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:52.441663 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" event={"ID":"1c3d2cac-d8c1-40e3-86b1-ae274f842e03","Type":"ContainerDied","Data":"f34f7fcdbcf90f16c38e81d0c65d9e5c27c51d1928880fa87711454971b96924"} Aug 05 17:04:52.442057 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:52.442043 2573 scope.go:117] "RemoveContainer" containerID="f34f7fcdbcf90f16c38e81d0c65d9e5c27c51d1928880fa87711454971b96924" Aug 05 17:04:52.443046 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:52.443015 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:04:52.443101 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:52.443086 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:04:52.443138 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:52.443119 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:04:53.451098 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:53.451052 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" event={"ID":"1c3d2cac-d8c1-40e3-86b1-ae274f842e03","Type":"ContainerStarted","Data":"085df308c92b1309c68c9265d469feb95f1e561df2234742f18cb0c3a29429e6"} Aug 05 17:04:54.364874 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:54.364829 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:54.365137 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:54.365112 2573 patch_prober.go:28] interesting pod/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Aug 05 17:04:54.365227 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:54.365169 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" podUID="1c3d2cac-d8c1-40e3-86b1-ae274f842e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Aug 05 17:04:55.364992 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:55.364950 2573 patch_prober.go:28] interesting pod/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Aug 05 17:04:55.365424 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:55.365028 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" podUID="1c3d2cac-d8c1-40e3-86b1-ae274f842e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Aug 05 17:04:56.365457 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:56.365410 2573 patch_prober.go:28] interesting pod/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Aug 05 17:04:56.365971 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:56.365518 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" podUID="1c3d2cac-d8c1-40e3-86b1-ae274f842e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Aug 05 17:04:57.365102 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:57.365061 2573 patch_prober.go:28] interesting pod/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Aug 05 17:04:57.365301 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:57.365123 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" podUID="1c3d2cac-d8c1-40e3-86b1-ae274f842e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Aug 05 17:04:58.365702 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:58.365657 2573 patch_prober.go:28] interesting pod/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Aug 05 17:04:58.366109 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:58.365750 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" podUID="1c3d2cac-d8c1-40e3-86b1-ae274f842e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Aug 05 17:04:59.369051 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:59.369023 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:59.369461 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:59.369273 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:04:59.370095 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:04:59.370078 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:05:02.165800 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:02.165766 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:02.753507 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:02.753475 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:02.858175 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:02.858138 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:03.059474 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.059440 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg"] Aug 05 17:05:03.059910 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.059896 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe1b190-54ab-4de3-ac85-bb73890d7d2c" containerName="istio-proxy" Aug 05 17:05:03.059973 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.059913 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe1b190-54ab-4de3-ac85-bb73890d7d2c" containerName="istio-proxy" Aug 05 17:05:03.060012 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.059989 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe1b190-54ab-4de3-ac85-bb73890d7d2c" containerName="istio-proxy" Aug 05 17:05:03.063131 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.063107 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.070911 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.070890 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-034ebc-openshift-default-dockercfg-6zj7q\"" Aug 05 17:05:03.091009 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.090983 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg"] Aug 05 17:05:03.156369 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.156326 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57ab2728-4202-4bb4-9b3e-e79efb629a62-istiod-ca-cert\") pod \"e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.156594 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.156394 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57ab2728-4202-4bb4-9b3e-e79efb629a62-istio-podinfo\") pod \"e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.156594 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.156429 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-workload-certs\") pod \"e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.156594 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.156452 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57ab2728-4202-4bb4-9b3e-e79efb629a62-istio-token\") pod \"e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.156594 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.156487 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-credential-socket\") pod \"e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.156594 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.156510 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-istio-envoy\") pod \"e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.156594 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.156565 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-workload-socket\") pod \"e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.156934 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.156599 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xnq6\" (UniqueName: \"kubernetes.io/projected/57ab2728-4202-4bb4-9b3e-e79efb629a62-kube-api-access-6xnq6\") pod \"e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.156934 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.156674 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-istio-data\") pod \"e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.238599 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.238568 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:03.257421 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.257394 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-workload-socket\") pod \"e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.257581 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.257428 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xnq6\" (UniqueName: \"kubernetes.io/projected/57ab2728-4202-4bb4-9b3e-e79efb629a62-kube-api-access-6xnq6\") pod \"e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.257581 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.257477 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-istio-data\") pod \"e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.257581 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.257509 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57ab2728-4202-4bb4-9b3e-e79efb629a62-istiod-ca-cert\") pod \"e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.257581 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.257527 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57ab2728-4202-4bb4-9b3e-e79efb629a62-istio-podinfo\") pod \"e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.257581 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.257555 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-workload-certs\") pod \"e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.257841 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.257825 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57ab2728-4202-4bb4-9b3e-e79efb629a62-istio-token\") pod \"e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.257892 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.257840 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-workload-socket\") pod \"e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.257892 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.257873 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-credential-socket\") pod \"e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.257892 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.257881 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-istio-data\") pod \"e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.258043 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.257902 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-istio-envoy\") pod \"e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.258043 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.257915 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-workload-certs\") pod \"e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.258160 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.258139 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-credential-socket\") pod \"e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.258447 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.258423 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57ab2728-4202-4bb4-9b3e-e79efb629a62-istiod-ca-cert\") pod \"e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.260028 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.260007 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57ab2728-4202-4bb4-9b3e-e79efb629a62-istio-podinfo\") pod \"e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.260127 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.260073 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-istio-envoy\") pod \"e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.297295 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.297267 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57ab2728-4202-4bb4-9b3e-e79efb629a62-istio-token\") pod \"e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.298505 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.298482 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xnq6\" (UniqueName: \"kubernetes.io/projected/57ab2728-4202-4bb4-9b3e-e79efb629a62-kube-api-access-6xnq6\") pod \"e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.373431 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.373326 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:03.390825 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.390800 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:03.509925 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.509902 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg"] Aug 05 17:05:03.511562 ip-10-0-132-37 kubenswrapper[2573]: W0805 17:05:03.511534 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ab2728_4202_4bb4_9b3e_e79efb629a62.slice/crio-766508afc8587f62ee1bdcd9ae8860568cc66683f7a87d66fb143314f11cc18b WatchSource:0}: Error finding container 766508afc8587f62ee1bdcd9ae8860568cc66683f7a87d66fb143314f11cc18b: Status 404 returned error can't find the container with id 766508afc8587f62ee1bdcd9ae8860568cc66683f7a87d66fb143314f11cc18b Aug 05 17:05:03.513656 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.513628 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:05:03.513724 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.513692 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:05:03.513724 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:03.513720 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:05:04.503116 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:04.503075 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" event={"ID":"57ab2728-4202-4bb4-9b3e-e79efb629a62","Type":"ContainerStarted","Data":"f244c624056f2b3d1342b7d74582d2dac154873e906c292d642d4be1b44ea7a5"} Aug 05 17:05:04.503553 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:04.503135 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" event={"ID":"57ab2728-4202-4bb4-9b3e-e79efb629a62","Type":"ContainerStarted","Data":"766508afc8587f62ee1bdcd9ae8860568cc66683f7a87d66fb143314f11cc18b"} Aug 05 17:05:04.527531 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:04.527486 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" podStartSLOduration=1.527470703 podStartE2EDuration="1.527470703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:05:04.526264688 +0000 UTC m=+2619.683110602" watchObservedRunningTime="2026-08-05 17:05:04.527470703 +0000 UTC m=+2619.684316653" Aug 05 17:05:05.373912 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:05.373881 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:05.379074 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:05.379046 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:05.507332 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:05.507298 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:05.508441 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:05.508423 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:06.533206 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:06.533173 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:06.625135 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:06.625105 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:07.760140 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:07.760108 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:07.920584 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:07.920545 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:08.425959 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:08.425926 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:08.638240 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:08.638193 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:21.093275 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:21.093238 2573 patch_prober.go:28] interesting pod/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": read tcp 10.133.0.2:58016->10.133.0.51:15021: read: connection reset by peer" start-of-body= Aug 05 17:05:21.093748 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:21.093306 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" podUID="57ab2728-4202-4bb4-9b3e-e79efb629a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": read tcp 10.133.0.2:58016->10.133.0.51:15021: read: connection reset by peer" Aug 05 17:05:21.585802 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:21.585769 2573 generic.go:356] "Generic (PLEG): container finished" podID="57ab2728-4202-4bb4-9b3e-e79efb629a62" containerID="f244c624056f2b3d1342b7d74582d2dac154873e906c292d642d4be1b44ea7a5" exitCode=0 Aug 05 17:05:21.585802 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:21.585806 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" event={"ID":"57ab2728-4202-4bb4-9b3e-e79efb629a62","Type":"ContainerDied","Data":"f244c624056f2b3d1342b7d74582d2dac154873e906c292d642d4be1b44ea7a5"} Aug 05 17:05:21.586158 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:21.586146 2573 scope.go:117] "RemoveContainer" containerID="f244c624056f2b3d1342b7d74582d2dac154873e906c292d642d4be1b44ea7a5" Aug 05 17:05:21.587146 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:21.587118 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:05:21.587249 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:21.587200 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:05:21.587311 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:21.587245 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:05:22.591502 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:22.591465 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" event={"ID":"57ab2728-4202-4bb4-9b3e-e79efb629a62","Type":"ContainerStarted","Data":"1c45d91b1155be11e946bba6ed8a88444e070cef153ca9932f851149b32ebfd8"} Aug 05 17:05:23.373936 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:23.373894 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:23.374153 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:23.374134 2573 patch_prober.go:28] interesting pod/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Aug 05 17:05:23.374231 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:23.374191 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" podUID="57ab2728-4202-4bb4-9b3e-e79efb629a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Aug 05 17:05:24.374518 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:24.374473 2573 patch_prober.go:28] interesting pod/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Aug 05 17:05:24.374938 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:24.374536 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" podUID="57ab2728-4202-4bb4-9b3e-e79efb629a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Aug 05 17:05:25.374410 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:25.374353 2573 patch_prober.go:28] interesting pod/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Aug 05 17:05:25.374601 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:25.374431 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" podUID="57ab2728-4202-4bb4-9b3e-e79efb629a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Aug 05 17:05:26.374149 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:26.374105 2573 patch_prober.go:28] interesting pod/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Aug 05 17:05:26.374430 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:26.374173 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" podUID="57ab2728-4202-4bb4-9b3e-e79efb629a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Aug 05 17:05:27.374260 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:27.374214 2573 patch_prober.go:28] interesting pod/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Aug 05 17:05:27.374792 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:27.374294 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" podUID="57ab2728-4202-4bb4-9b3e-e79efb629a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Aug 05 17:05:28.377390 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:28.377351 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:28.377846 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:28.377635 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:28.378265 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:28.378248 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:05:31.437577 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:31.437544 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:39.704345 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:39.704308 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:39.854304 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:39.854272 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:42.094814 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.094785 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5"] Aug 05 17:05:42.098107 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.098091 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" Aug 05 17:05:42.102580 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.102556 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-09f3f8\"/\"test-model-09f3f8-kserve-self-signed-certs\"" Aug 05 17:05:42.102580 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.102579 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-09f3f8\"/\"kube-root-ca.crt\"" Aug 05 17:05:42.102768 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.102578 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-09f3f8\"/\"openshift-service-ca.crt\"" Aug 05 17:05:42.102768 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.102559 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-09f3f8\"/\"default-dockercfg-fcltp\"" Aug 05 17:05:42.108772 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.108750 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5"] Aug 05 17:05:42.126221 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.126196 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:42.206711 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.206673 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqpc5\" (UniqueName: \"kubernetes.io/projected/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-kube-api-access-lqpc5\") pod \"test-model-09f3f8-kserve-844cf486fc-v57n5\" (UID: \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\") " pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" Aug 05 17:05:42.206878 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.206726 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-tls-certs\") pod \"test-model-09f3f8-kserve-844cf486fc-v57n5\" (UID: \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\") " pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" Aug 05 17:05:42.206878 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.206779 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-dshm\") pod \"test-model-09f3f8-kserve-844cf486fc-v57n5\" (UID: \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\") " pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" Aug 05 17:05:42.206878 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.206796 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-model-cache\") pod \"test-model-09f3f8-kserve-844cf486fc-v57n5\" (UID: \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\") " pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" Aug 05 17:05:42.206981 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.206897 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-tmp-dir\") pod \"test-model-09f3f8-kserve-844cf486fc-v57n5\" (UID: \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\") " pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" Aug 05 17:05:42.206981 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.206934 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-home\") pod \"test-model-09f3f8-kserve-844cf486fc-v57n5\" (UID: \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\") " pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" Aug 05 17:05:42.307688 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.307650 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-tmp-dir\") pod \"test-model-09f3f8-kserve-844cf486fc-v57n5\" (UID: \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\") " pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" Aug 05 17:05:42.307871 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.307693 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-home\") pod \"test-model-09f3f8-kserve-844cf486fc-v57n5\" (UID: \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\") " pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" Aug 05 17:05:42.307871 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.307734 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqpc5\" (UniqueName: \"kubernetes.io/projected/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-kube-api-access-lqpc5\") pod \"test-model-09f3f8-kserve-844cf486fc-v57n5\" (UID: \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\") " pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" Aug 05 17:05:42.307871 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.307762 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-tls-certs\") pod \"test-model-09f3f8-kserve-844cf486fc-v57n5\" (UID: \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\") " pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" Aug 05 17:05:42.307871 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.307789 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-dshm\") pod \"test-model-09f3f8-kserve-844cf486fc-v57n5\" (UID: \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\") " pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" Aug 05 17:05:42.307871 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.307804 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-model-cache\") pod \"test-model-09f3f8-kserve-844cf486fc-v57n5\" (UID: \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\") " pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" Aug 05 17:05:42.308148 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.308131 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-tmp-dir\") pod \"test-model-09f3f8-kserve-844cf486fc-v57n5\" (UID: \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\") " pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" Aug 05 17:05:42.308210 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.308151 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-home\") pod \"test-model-09f3f8-kserve-844cf486fc-v57n5\" (UID: \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\") " pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" Aug 05 17:05:42.308210 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.308170 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-model-cache\") pod \"test-model-09f3f8-kserve-844cf486fc-v57n5\" (UID: \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\") " pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" Aug 05 17:05:42.310126 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.310096 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-dshm\") pod \"test-model-09f3f8-kserve-844cf486fc-v57n5\" (UID: \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\") " pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" Aug 05 17:05:42.310236 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.310221 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-tls-certs\") pod \"test-model-09f3f8-kserve-844cf486fc-v57n5\" (UID: \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\") " pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" Aug 05 17:05:42.317525 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.317495 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqpc5\" (UniqueName: \"kubernetes.io/projected/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-kube-api-access-lqpc5\") pod \"test-model-09f3f8-kserve-844cf486fc-v57n5\" (UID: \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\") " pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" Aug 05 17:05:42.407695 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.407599 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" Aug 05 17:05:42.542769 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.542742 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5"] Aug 05 17:05:42.544598 ip-10-0-132-37 kubenswrapper[2573]: W0805 17:05:42.544539 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fbde41b_4e0e_46a3_8d9b_2a523cfa0896.slice/crio-b4a01cdd2f66dda3a353505682b10c1523040e6bb3c3e140d02c4412ad0ce5e8 WatchSource:0}: Error finding container b4a01cdd2f66dda3a353505682b10c1523040e6bb3c3e140d02c4412ad0ce5e8: Status 404 returned error can't find the container with id b4a01cdd2f66dda3a353505682b10c1523040e6bb3c3e140d02c4412ad0ce5e8 Aug 05 17:05:42.680958 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.680885 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" event={"ID":"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896","Type":"ContainerStarted","Data":"53767668a02dd26cae18a64f968330477dc4d1b431aef5e9e2217983159b565b"} Aug 05 17:05:42.680958 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.680923 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" event={"ID":"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896","Type":"ContainerStarted","Data":"b4a01cdd2f66dda3a353505682b10c1523040e6bb3c3e140d02c4412ad0ce5e8"} Aug 05 17:05:42.705609 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:42.705562 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" podStartSLOduration=0.705545581 podStartE2EDuration="705.545581ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:05:42.703981449 +0000 UTC m=+2657.860827349" watchObservedRunningTime="2026-08-05 17:05:42.705545581 +0000 UTC m=+2657.862391482" Aug 05 17:05:52.080341 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:52.080289 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:05:52.408719 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:52.408620 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" Aug 05 17:05:52.408719 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:52.408659 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" Aug 05 17:05:52.421338 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:52.421304 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" Aug 05 17:05:52.738983 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:05:52.738896 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" Aug 05 17:06:06.248055 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:06:06.248020 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:06:51.250486 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:06:51.250447 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5"] Aug 05 17:06:51.286544 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:06:51.286501 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:06:52.809671 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:06:52.809636 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:02.871550 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:02.871515 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:02.975977 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:02.975948 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:03.071626 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:03.071593 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:03.168067 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:03.167981 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:03.667719 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:03.667684 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:06.269150 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:06.269103 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" podUID="5fbde41b-4e0e-46a3-8d9b-2a523cfa0896" containerName="main" containerID="cri-o://53767668a02dd26cae18a64f968330477dc4d1b431aef5e9e2217983159b565b" gracePeriod=45 Aug 05 17:07:06.513991 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:06.513965 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" Aug 05 17:07:06.545766 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:06.545691 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-tmp-dir\") pod \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\" (UID: \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\") " Aug 05 17:07:06.545766 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:06.545742 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-home\") pod \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\" (UID: \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\") " Aug 05 17:07:06.545983 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:06.545770 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-model-cache\") pod \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\" (UID: \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\") " Aug 05 17:07:06.545983 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:06.545850 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-dshm\") pod \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\" (UID: \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\") " Aug 05 17:07:06.545983 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:06.545882 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-tls-certs\") pod \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\" (UID: \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\") " Aug 05 17:07:06.545983 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:06.545960 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqpc5\" (UniqueName: \"kubernetes.io/projected/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-kube-api-access-lqpc5\") pod \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\" (UID: \"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896\") " Aug 05 17:07:06.546196 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:06.545999 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5fbde41b-4e0e-46a3-8d9b-2a523cfa0896" (UID: "5fbde41b-4e0e-46a3-8d9b-2a523cfa0896"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:07:06.546196 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:06.546079 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-home" (OuterVolumeSpecName: "home") pod "5fbde41b-4e0e-46a3-8d9b-2a523cfa0896" (UID: "5fbde41b-4e0e-46a3-8d9b-2a523cfa0896"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:07:06.546196 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:06.546135 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-model-cache" (OuterVolumeSpecName: "model-cache") pod "5fbde41b-4e0e-46a3-8d9b-2a523cfa0896" (UID: "5fbde41b-4e0e-46a3-8d9b-2a523cfa0896"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:07:06.546344 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:06.546272 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-tmp-dir\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:07:06.546344 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:06.546290 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-home\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:07:06.546344 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:06.546304 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-model-cache\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:07:06.548144 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:06.548125 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5fbde41b-4e0e-46a3-8d9b-2a523cfa0896" (UID: "5fbde41b-4e0e-46a3-8d9b-2a523cfa0896"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:07:06.548430 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:06.548412 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-dshm" (OuterVolumeSpecName: "dshm") pod "5fbde41b-4e0e-46a3-8d9b-2a523cfa0896" (UID: "5fbde41b-4e0e-46a3-8d9b-2a523cfa0896"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:07:06.548503 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:06.548452 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-kube-api-access-lqpc5" (OuterVolumeSpecName: "kube-api-access-lqpc5") pod "5fbde41b-4e0e-46a3-8d9b-2a523cfa0896" (UID: "5fbde41b-4e0e-46a3-8d9b-2a523cfa0896"). InnerVolumeSpecName "kube-api-access-lqpc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:07:06.647329 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:06.647292 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-dshm\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:07:06.647329 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:06.647322 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-tls-certs\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:07:06.647329 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:06.647334 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqpc5\" (UniqueName: \"kubernetes.io/projected/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896-kube-api-access-lqpc5\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:07:07.059869 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:07.059835 2573 generic.go:356] "Generic (PLEG): container finished" podID="5fbde41b-4e0e-46a3-8d9b-2a523cfa0896" containerID="53767668a02dd26cae18a64f968330477dc4d1b431aef5e9e2217983159b565b" exitCode=0 Aug 05 17:07:07.060044 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:07.059881 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" Aug 05 17:07:07.060044 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:07.059925 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" event={"ID":"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896","Type":"ContainerDied","Data":"53767668a02dd26cae18a64f968330477dc4d1b431aef5e9e2217983159b565b"} Aug 05 17:07:07.060044 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:07.059971 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5" event={"ID":"5fbde41b-4e0e-46a3-8d9b-2a523cfa0896","Type":"ContainerDied","Data":"b4a01cdd2f66dda3a353505682b10c1523040e6bb3c3e140d02c4412ad0ce5e8"} Aug 05 17:07:07.060044 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:07.059990 2573 scope.go:117] "RemoveContainer" containerID="53767668a02dd26cae18a64f968330477dc4d1b431aef5e9e2217983159b565b" Aug 05 17:07:07.068926 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:07.068909 2573 scope.go:117] "RemoveContainer" containerID="53767668a02dd26cae18a64f968330477dc4d1b431aef5e9e2217983159b565b" Aug 05 17:07:07.069160 ip-10-0-132-37 kubenswrapper[2573]: E0805 17:07:07.069143 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53767668a02dd26cae18a64f968330477dc4d1b431aef5e9e2217983159b565b\": container with ID starting with 53767668a02dd26cae18a64f968330477dc4d1b431aef5e9e2217983159b565b not found: ID does not exist" containerID="53767668a02dd26cae18a64f968330477dc4d1b431aef5e9e2217983159b565b" Aug 05 17:07:07.069227 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:07.069173 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53767668a02dd26cae18a64f968330477dc4d1b431aef5e9e2217983159b565b"} err="failed to get container status \"53767668a02dd26cae18a64f968330477dc4d1b431aef5e9e2217983159b565b\": rpc error: code = NotFound desc = could not find container \"53767668a02dd26cae18a64f968330477dc4d1b431aef5e9e2217983159b565b\": container with ID starting with 53767668a02dd26cae18a64f968330477dc4d1b431aef5e9e2217983159b565b not found: ID does not exist" Aug 05 17:07:07.083534 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:07.083507 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5"] Aug 05 17:07:07.091012 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:07.090990 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-09f3f8/test-model-09f3f8-kserve-844cf486fc-v57n5"] Aug 05 17:07:07.303400 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:07.303347 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fbde41b-4e0e-46a3-8d9b-2a523cfa0896" path="/var/lib/kubelet/pods/5fbde41b-4e0e-46a3-8d9b-2a523cfa0896/volumes" Aug 05 17:07:10.127423 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:10.127364 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:10.217111 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:10.217076 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:10.358432 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:10.358399 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc"] Aug 05 17:07:10.358690 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:10.358665 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" podUID="1c3d2cac-d8c1-40e3-86b1-ae274f842e03" containerName="istio-proxy" containerID="cri-o://085df308c92b1309c68c9265d469feb95f1e561df2234742f18cb0c3a29429e6" gracePeriod=30 Aug 05 17:07:10.389288 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:10.389174 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:10.787642 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:10.787613 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:14.372231 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:14.372190 2573 patch_prober.go:28] interesting pod/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 05 17:07:14.372690 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:14.372260 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" podUID="1c3d2cac-d8c1-40e3-86b1-ae274f842e03" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 05 17:07:15.600419 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.600394 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:07:15.723509 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.723417 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8bvr\" (UniqueName: \"kubernetes.io/projected/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-kube-api-access-n8bvr\") pod \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " Aug 05 17:07:15.723509 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.723478 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istio-token\") pod \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " Aug 05 17:07:15.723509 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.723506 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istiod-ca-cert\") pod \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " Aug 05 17:07:15.723814 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.723534 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istio-data\") pod \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " Aug 05 17:07:15.723814 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.723624 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istio-envoy\") pod \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " Aug 05 17:07:15.723814 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.723660 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-credential-socket\") pod \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " Aug 05 17:07:15.723814 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.723703 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-workload-certs\") pod \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " Aug 05 17:07:15.723814 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.723738 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istio-podinfo\") pod \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " Aug 05 17:07:15.723814 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.723804 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-workload-socket\") pod \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\" (UID: \"1c3d2cac-d8c1-40e3-86b1-ae274f842e03\") " Aug 05 17:07:15.725109 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.723975 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1c3d2cac-d8c1-40e3-86b1-ae274f842e03" (UID: "1c3d2cac-d8c1-40e3-86b1-ae274f842e03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:07:15.725109 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.724044 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istio-data" (OuterVolumeSpecName: "istio-data") pod "1c3d2cac-d8c1-40e3-86b1-ae274f842e03" (UID: "1c3d2cac-d8c1-40e3-86b1-ae274f842e03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:07:15.725109 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.723968 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1c3d2cac-d8c1-40e3-86b1-ae274f842e03" (UID: "1c3d2cac-d8c1-40e3-86b1-ae274f842e03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:07:15.725109 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.724160 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1c3d2cac-d8c1-40e3-86b1-ae274f842e03" (UID: "1c3d2cac-d8c1-40e3-86b1-ae274f842e03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:07:15.725109 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.724405 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-credential-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:07:15.725109 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.724423 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-workload-certs\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:07:15.725109 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.724438 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istiod-ca-cert\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:07:15.725109 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.724454 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istio-data\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:07:15.725109 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.724504 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1c3d2cac-d8c1-40e3-86b1-ae274f842e03" (UID: "1c3d2cac-d8c1-40e3-86b1-ae274f842e03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:07:15.726583 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.726546 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1c3d2cac-d8c1-40e3-86b1-ae274f842e03" (UID: "1c3d2cac-d8c1-40e3-86b1-ae274f842e03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 17:07:15.728113 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.728089 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-kube-api-access-n8bvr" (OuterVolumeSpecName: "kube-api-access-n8bvr") pod "1c3d2cac-d8c1-40e3-86b1-ae274f842e03" (UID: "1c3d2cac-d8c1-40e3-86b1-ae274f842e03"). InnerVolumeSpecName "kube-api-access-n8bvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:07:15.729874 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.729849 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1c3d2cac-d8c1-40e3-86b1-ae274f842e03" (UID: "1c3d2cac-d8c1-40e3-86b1-ae274f842e03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:07:15.729874 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.729858 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istio-token" (OuterVolumeSpecName: "istio-token") pod "1c3d2cac-d8c1-40e3-86b1-ae274f842e03" (UID: "1c3d2cac-d8c1-40e3-86b1-ae274f842e03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:07:15.793208 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.793172 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:15.829980 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.825974 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-workload-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:07:15.829980 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.826008 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8bvr\" (UniqueName: \"kubernetes.io/projected/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-kube-api-access-n8bvr\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:07:15.829980 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.826027 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istio-token\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:07:15.829980 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.826040 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istio-envoy\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:07:15.829980 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.826059 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c3d2cac-d8c1-40e3-86b1-ae274f842e03-istio-podinfo\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:07:15.892564 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.892523 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:15.985863 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:15.985783 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:16.102739 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:16.102706 2573 generic.go:356] "Generic (PLEG): container finished" podID="1c3d2cac-d8c1-40e3-86b1-ae274f842e03" containerID="085df308c92b1309c68c9265d469feb95f1e561df2234742f18cb0c3a29429e6" exitCode=0 Aug 05 17:07:16.102912 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:16.102765 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" Aug 05 17:07:16.102912 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:16.102781 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" event={"ID":"1c3d2cac-d8c1-40e3-86b1-ae274f842e03","Type":"ContainerDied","Data":"085df308c92b1309c68c9265d469feb95f1e561df2234742f18cb0c3a29429e6"} Aug 05 17:07:16.102912 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:16.102856 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc" event={"ID":"1c3d2cac-d8c1-40e3-86b1-ae274f842e03","Type":"ContainerDied","Data":"72035fbde5023b8d30a485ecc0773edf927ef1d47f47c525e503c24b624e3edb"} Aug 05 17:07:16.102912 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:16.102878 2573 scope.go:117] "RemoveContainer" containerID="085df308c92b1309c68c9265d469feb95f1e561df2234742f18cb0c3a29429e6" Aug 05 17:07:16.111829 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:16.111812 2573 scope.go:117] "RemoveContainer" containerID="f34f7fcdbcf90f16c38e81d0c65d9e5c27c51d1928880fa87711454971b96924" Aug 05 17:07:16.119455 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:16.119436 2573 scope.go:117] "RemoveContainer" containerID="085df308c92b1309c68c9265d469feb95f1e561df2234742f18cb0c3a29429e6" Aug 05 17:07:16.119688 ip-10-0-132-37 kubenswrapper[2573]: E0805 17:07:16.119672 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"085df308c92b1309c68c9265d469feb95f1e561df2234742f18cb0c3a29429e6\": container with ID starting with 085df308c92b1309c68c9265d469feb95f1e561df2234742f18cb0c3a29429e6 not found: ID does not exist" containerID="085df308c92b1309c68c9265d469feb95f1e561df2234742f18cb0c3a29429e6" Aug 05 17:07:16.119734 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:16.119696 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"085df308c92b1309c68c9265d469feb95f1e561df2234742f18cb0c3a29429e6"} err="failed to get container status \"085df308c92b1309c68c9265d469feb95f1e561df2234742f18cb0c3a29429e6\": rpc error: code = NotFound desc = could not find container \"085df308c92b1309c68c9265d469feb95f1e561df2234742f18cb0c3a29429e6\": container with ID starting with 085df308c92b1309c68c9265d469feb95f1e561df2234742f18cb0c3a29429e6 not found: ID does not exist" Aug 05 17:07:16.119734 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:16.119713 2573 scope.go:117] "RemoveContainer" containerID="f34f7fcdbcf90f16c38e81d0c65d9e5c27c51d1928880fa87711454971b96924" Aug 05 17:07:16.119907 ip-10-0-132-37 kubenswrapper[2573]: E0805 17:07:16.119891 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f34f7fcdbcf90f16c38e81d0c65d9e5c27c51d1928880fa87711454971b96924\": container with ID starting with f34f7fcdbcf90f16c38e81d0c65d9e5c27c51d1928880fa87711454971b96924 not found: ID does not exist" containerID="f34f7fcdbcf90f16c38e81d0c65d9e5c27c51d1928880fa87711454971b96924" Aug 05 17:07:16.119944 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:16.119911 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f34f7fcdbcf90f16c38e81d0c65d9e5c27c51d1928880fa87711454971b96924"} err="failed to get container status \"f34f7fcdbcf90f16c38e81d0c65d9e5c27c51d1928880fa87711454971b96924\": rpc error: code = NotFound desc = could not find container \"f34f7fcdbcf90f16c38e81d0c65d9e5c27c51d1928880fa87711454971b96924\": container with ID starting with f34f7fcdbcf90f16c38e81d0c65d9e5c27c51d1928880fa87711454971b96924 not found: ID does not exist" Aug 05 17:07:16.128419 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:16.128397 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc"] Aug 05 17:07:16.133973 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:16.133953 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-09f3f8-openshift-default-748f49bc5d-9p9qc"] Aug 05 17:07:16.317346 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:16.317314 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:17.304868 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:17.304836 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c3d2cac-d8c1-40e3-86b1-ae274f842e03" path="/var/lib/kubelet/pods/1c3d2cac-d8c1-40e3-86b1-ae274f842e03/volumes" Aug 05 17:07:22.435542 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:22.435506 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:22.536086 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:22.536054 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:22.671323 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:22.671292 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg"] Aug 05 17:07:22.671560 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:22.671536 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" podUID="57ab2728-4202-4bb4-9b3e-e79efb629a62" containerName="istio-proxy" containerID="cri-o://1c45d91b1155be11e946bba6ed8a88444e070cef153ca9932f851149b32ebfd8" gracePeriod=30 Aug 05 17:07:22.709080 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:22.709001 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:22.917792 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:22.917757 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:23.262715 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.262681 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:23.532107 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.532021 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478"] Aug 05 17:07:23.532576 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.532345 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c3d2cac-d8c1-40e3-86b1-ae274f842e03" containerName="istio-proxy" Aug 05 17:07:23.532576 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.532356 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c3d2cac-d8c1-40e3-86b1-ae274f842e03" containerName="istio-proxy" Aug 05 17:07:23.532576 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.532369 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fbde41b-4e0e-46a3-8d9b-2a523cfa0896" containerName="main" Aug 05 17:07:23.532576 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.532390 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fbde41b-4e0e-46a3-8d9b-2a523cfa0896" containerName="main" Aug 05 17:07:23.532576 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.532401 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c3d2cac-d8c1-40e3-86b1-ae274f842e03" containerName="istio-proxy" Aug 05 17:07:23.532576 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.532406 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c3d2cac-d8c1-40e3-86b1-ae274f842e03" containerName="istio-proxy" Aug 05 17:07:23.532576 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.532460 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c3d2cac-d8c1-40e3-86b1-ae274f842e03" containerName="istio-proxy" Aug 05 17:07:23.532576 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.532467 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fbde41b-4e0e-46a3-8d9b-2a523cfa0896" containerName="main" Aug 05 17:07:23.532576 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.532474 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c3d2cac-d8c1-40e3-86b1-ae274f842e03" containerName="istio-proxy" Aug 05 17:07:23.536864 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.536839 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.540512 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.540489 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-64d8dc-openshift-default-dockercfg-k6tbb\"" Aug 05 17:07:23.549886 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.549862 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478"] Aug 05 17:07:23.578403 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.578348 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:23.694236 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.694196 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff7p2\" (UniqueName: \"kubernetes.io/projected/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-kube-api-access-ff7p2\") pod \"e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.694472 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.694249 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-workload-certs\") pod \"e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.694472 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.694282 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-workload-socket\") pod \"e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.694472 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.694304 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istio-podinfo\") pod \"e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.694472 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.694343 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-credential-socket\") pod \"e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.694472 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.694402 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istio-token\") pod \"e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.694472 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.694453 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istio-envoy\") pod \"e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.694740 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.694480 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istiod-ca-cert\") pod \"e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.694740 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.694535 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istio-data\") pod \"e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.753824 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.753793 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:23.795817 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.795724 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istio-envoy\") pod \"e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.795817 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.795758 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istiod-ca-cert\") pod \"e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.796042 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.795887 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istio-data\") pod \"e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.796042 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.795933 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff7p2\" (UniqueName: \"kubernetes.io/projected/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-kube-api-access-ff7p2\") pod \"e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.796042 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.795975 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-workload-certs\") pod \"e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.796042 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.796001 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-workload-socket\") pod \"e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.796042 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.796016 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istio-podinfo\") pod \"e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.796309 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.796234 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-credential-socket\") pod \"e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.796309 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.796271 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istio-token\") pod \"e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.796309 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.796281 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istio-data\") pod \"e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.796545 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.796520 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-workload-certs\") pod \"e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.796759 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.796729 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-workload-socket\") pod \"e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.796881 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.796744 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istiod-ca-cert\") pod \"e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.796881 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.796790 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-credential-socket\") pod \"e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.798481 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.798461 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istio-envoy\") pod \"e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.798610 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.798593 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istio-podinfo\") pod \"e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.805831 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.805804 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istio-token\") pod \"e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.806005 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.805987 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff7p2\" (UniqueName: \"kubernetes.io/projected/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-kube-api-access-ff7p2\") pod \"e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.849913 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.849882 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:23.989429 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.989400 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478"] Aug 05 17:07:23.991457 ip-10-0-132-37 kubenswrapper[2573]: W0805 17:07:23.991428 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e0415c6_1bd9_4cf1_9755_9c6b0e8cec9a.slice/crio-df729bbdef34fd511c3f61a95c6400e4592678a871c5384719a97145b57b0296 WatchSource:0}: Error finding container df729bbdef34fd511c3f61a95c6400e4592678a871c5384719a97145b57b0296: Status 404 returned error can't find the container with id df729bbdef34fd511c3f61a95c6400e4592678a871c5384719a97145b57b0296 Aug 05 17:07:23.993707 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.993671 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:07:23.993776 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.993755 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:07:23.993816 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:23.993799 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:07:24.143852 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:24.143771 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" event={"ID":"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a","Type":"ContainerStarted","Data":"83ff39e22a9d2f165db6c57536e8bd67413cabf65be9c3845c3520f66f254ddc"} Aug 05 17:07:24.144043 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:24.143881 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" event={"ID":"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a","Type":"ContainerStarted","Data":"df729bbdef34fd511c3f61a95c6400e4592678a871c5384719a97145b57b0296"} Aug 05 17:07:24.168589 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:24.168535 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" podStartSLOduration=1.168515093 podStartE2EDuration="1.168515093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:07:24.166577943 +0000 UTC m=+2759.323423884" watchObservedRunningTime="2026-08-05 17:07:24.168515093 +0000 UTC m=+2759.325360995" Aug 05 17:07:24.850325 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:24.850286 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:25.855774 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:25.855746 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:26.091563 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:26.091510 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:26.153752 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:26.153721 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:26.154586 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:26.154567 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:26.187881 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:26.187850 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:27.314031 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:27.313997 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:27.656735 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:27.656653 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:27.906013 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:27.905986 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:27.919463 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:27.919388 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:07:28.005817 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.005780 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:28.035902 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.035867 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57ab2728-4202-4bb4-9b3e-e79efb629a62-istio-podinfo\") pod \"57ab2728-4202-4bb4-9b3e-e79efb629a62\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " Aug 05 17:07:28.036104 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.035930 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57ab2728-4202-4bb4-9b3e-e79efb629a62-istio-token\") pod \"57ab2728-4202-4bb4-9b3e-e79efb629a62\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " Aug 05 17:07:28.036104 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.035953 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57ab2728-4202-4bb4-9b3e-e79efb629a62-istiod-ca-cert\") pod \"57ab2728-4202-4bb4-9b3e-e79efb629a62\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " Aug 05 17:07:28.036104 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.035992 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xnq6\" (UniqueName: \"kubernetes.io/projected/57ab2728-4202-4bb4-9b3e-e79efb629a62-kube-api-access-6xnq6\") pod \"57ab2728-4202-4bb4-9b3e-e79efb629a62\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " Aug 05 17:07:28.036104 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.036018 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-workload-certs\") pod \"57ab2728-4202-4bb4-9b3e-e79efb629a62\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " Aug 05 17:07:28.036104 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.036043 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-istio-data\") pod \"57ab2728-4202-4bb4-9b3e-e79efb629a62\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " Aug 05 17:07:28.036104 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.036065 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-credential-socket\") pod \"57ab2728-4202-4bb4-9b3e-e79efb629a62\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " Aug 05 17:07:28.036104 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.036088 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-workload-socket\") pod \"57ab2728-4202-4bb4-9b3e-e79efb629a62\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " Aug 05 17:07:28.036503 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.036157 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-istio-envoy\") pod \"57ab2728-4202-4bb4-9b3e-e79efb629a62\" (UID: \"57ab2728-4202-4bb4-9b3e-e79efb629a62\") " Aug 05 17:07:28.036503 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.036440 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57ab2728-4202-4bb4-9b3e-e79efb629a62-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "57ab2728-4202-4bb4-9b3e-e79efb629a62" (UID: "57ab2728-4202-4bb4-9b3e-e79efb629a62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:07:28.036701 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.036679 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "57ab2728-4202-4bb4-9b3e-e79efb629a62" (UID: "57ab2728-4202-4bb4-9b3e-e79efb629a62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:07:28.036794 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.036770 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "57ab2728-4202-4bb4-9b3e-e79efb629a62" (UID: "57ab2728-4202-4bb4-9b3e-e79efb629a62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:07:28.036875 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.036851 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "57ab2728-4202-4bb4-9b3e-e79efb629a62" (UID: "57ab2728-4202-4bb4-9b3e-e79efb629a62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:07:28.036875 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.036863 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-istio-data" (OuterVolumeSpecName: "istio-data") pod "57ab2728-4202-4bb4-9b3e-e79efb629a62" (UID: "57ab2728-4202-4bb4-9b3e-e79efb629a62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:07:28.038484 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.038460 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ab2728-4202-4bb4-9b3e-e79efb629a62-istio-token" (OuterVolumeSpecName: "istio-token") pod "57ab2728-4202-4bb4-9b3e-e79efb629a62" (UID: "57ab2728-4202-4bb4-9b3e-e79efb629a62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:07:28.038721 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.038681 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57ab2728-4202-4bb4-9b3e-e79efb629a62-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "57ab2728-4202-4bb4-9b3e-e79efb629a62" (UID: "57ab2728-4202-4bb4-9b3e-e79efb629a62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 17:07:28.039201 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.039179 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ab2728-4202-4bb4-9b3e-e79efb629a62-kube-api-access-6xnq6" (OuterVolumeSpecName: "kube-api-access-6xnq6") pod "57ab2728-4202-4bb4-9b3e-e79efb629a62" (UID: "57ab2728-4202-4bb4-9b3e-e79efb629a62"). InnerVolumeSpecName "kube-api-access-6xnq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:07:28.039305 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.039281 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "57ab2728-4202-4bb4-9b3e-e79efb629a62" (UID: "57ab2728-4202-4bb4-9b3e-e79efb629a62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:07:28.137132 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.137095 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-workload-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:07:28.137132 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.137126 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-istio-envoy\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:07:28.137132 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.137136 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57ab2728-4202-4bb4-9b3e-e79efb629a62-istio-podinfo\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:07:28.137132 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.137145 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57ab2728-4202-4bb4-9b3e-e79efb629a62-istio-token\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:07:28.137489 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.137153 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57ab2728-4202-4bb4-9b3e-e79efb629a62-istiod-ca-cert\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:07:28.137489 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.137161 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xnq6\" (UniqueName: \"kubernetes.io/projected/57ab2728-4202-4bb4-9b3e-e79efb629a62-kube-api-access-6xnq6\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:07:28.137489 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.137172 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-workload-certs\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:07:28.137489 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.137180 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-istio-data\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:07:28.137489 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.137190 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ab2728-4202-4bb4-9b3e-e79efb629a62-credential-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:07:28.164699 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.164661 2573 generic.go:356] "Generic (PLEG): container finished" podID="57ab2728-4202-4bb4-9b3e-e79efb629a62" containerID="1c45d91b1155be11e946bba6ed8a88444e070cef153ca9932f851149b32ebfd8" exitCode=0 Aug 05 17:07:28.164855 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.164712 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" Aug 05 17:07:28.164855 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.164737 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" event={"ID":"57ab2728-4202-4bb4-9b3e-e79efb629a62","Type":"ContainerDied","Data":"1c45d91b1155be11e946bba6ed8a88444e070cef153ca9932f851149b32ebfd8"} Aug 05 17:07:28.164855 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.164795 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg" event={"ID":"57ab2728-4202-4bb4-9b3e-e79efb629a62","Type":"ContainerDied","Data":"766508afc8587f62ee1bdcd9ae8860568cc66683f7a87d66fb143314f11cc18b"} Aug 05 17:07:28.164855 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.164818 2573 scope.go:117] "RemoveContainer" containerID="1c45d91b1155be11e946bba6ed8a88444e070cef153ca9932f851149b32ebfd8" Aug 05 17:07:28.175418 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.175396 2573 scope.go:117] "RemoveContainer" containerID="f244c624056f2b3d1342b7d74582d2dac154873e906c292d642d4be1b44ea7a5" Aug 05 17:07:28.183310 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.183291 2573 scope.go:117] "RemoveContainer" containerID="1c45d91b1155be11e946bba6ed8a88444e070cef153ca9932f851149b32ebfd8" Aug 05 17:07:28.183634 ip-10-0-132-37 kubenswrapper[2573]: E0805 17:07:28.183615 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c45d91b1155be11e946bba6ed8a88444e070cef153ca9932f851149b32ebfd8\": container with ID starting with 1c45d91b1155be11e946bba6ed8a88444e070cef153ca9932f851149b32ebfd8 not found: ID does not exist" containerID="1c45d91b1155be11e946bba6ed8a88444e070cef153ca9932f851149b32ebfd8" Aug 05 17:07:28.183690 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.183646 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c45d91b1155be11e946bba6ed8a88444e070cef153ca9932f851149b32ebfd8"} err="failed to get container status \"1c45d91b1155be11e946bba6ed8a88444e070cef153ca9932f851149b32ebfd8\": rpc error: code = NotFound desc = could not find container \"1c45d91b1155be11e946bba6ed8a88444e070cef153ca9932f851149b32ebfd8\": container with ID starting with 1c45d91b1155be11e946bba6ed8a88444e070cef153ca9932f851149b32ebfd8 not found: ID does not exist" Aug 05 17:07:28.183690 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.183663 2573 scope.go:117] "RemoveContainer" containerID="f244c624056f2b3d1342b7d74582d2dac154873e906c292d642d4be1b44ea7a5" Aug 05 17:07:28.183945 ip-10-0-132-37 kubenswrapper[2573]: E0805 17:07:28.183928 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f244c624056f2b3d1342b7d74582d2dac154873e906c292d642d4be1b44ea7a5\": container with ID starting with f244c624056f2b3d1342b7d74582d2dac154873e906c292d642d4be1b44ea7a5 not found: ID does not exist" containerID="f244c624056f2b3d1342b7d74582d2dac154873e906c292d642d4be1b44ea7a5" Aug 05 17:07:28.183995 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.183951 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f244c624056f2b3d1342b7d74582d2dac154873e906c292d642d4be1b44ea7a5"} err="failed to get container status \"f244c624056f2b3d1342b7d74582d2dac154873e906c292d642d4be1b44ea7a5\": rpc error: code = NotFound desc = could not find container \"f244c624056f2b3d1342b7d74582d2dac154873e906c292d642d4be1b44ea7a5\": container with ID starting with f244c624056f2b3d1342b7d74582d2dac154873e906c292d642d4be1b44ea7a5 not found: ID does not exist" Aug 05 17:07:28.192064 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.192035 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg"] Aug 05 17:07:28.194225 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:28.194199 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-034ebc-openshift-default-9f79d5774-58xvg"] Aug 05 17:07:29.303892 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:29.303860 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57ab2728-4202-4bb4-9b3e-e79efb629a62" path="/var/lib/kubelet/pods/57ab2728-4202-4bb4-9b3e-e79efb629a62/volumes" Aug 05 17:07:41.155056 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:41.155019 2573 patch_prober.go:28] interesting pod/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Aug 05 17:07:41.155501 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:41.155104 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" podUID="1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Aug 05 17:07:41.223983 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:41.223948 2573 generic.go:356] "Generic (PLEG): container finished" podID="1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a" containerID="83ff39e22a9d2f165db6c57536e8bd67413cabf65be9c3845c3520f66f254ddc" exitCode=0 Aug 05 17:07:41.223983 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:41.223988 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" event={"ID":"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a","Type":"ContainerDied","Data":"83ff39e22a9d2f165db6c57536e8bd67413cabf65be9c3845c3520f66f254ddc"} Aug 05 17:07:41.224471 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:41.224451 2573 scope.go:117] "RemoveContainer" containerID="83ff39e22a9d2f165db6c57536e8bd67413cabf65be9c3845c3520f66f254ddc" Aug 05 17:07:41.225548 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:41.225518 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:07:41.225623 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:41.225607 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:07:41.225662 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:41.225644 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:07:42.231873 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:42.231833 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" event={"ID":"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a","Type":"ContainerStarted","Data":"019d0fa9cf6d356475c42a825029a80b88ef4699448b9bf783fdf72f6158f031"} Aug 05 17:07:42.850445 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:42.850394 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:42.850829 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:42.850793 2573 patch_prober.go:28] interesting pod/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Aug 05 17:07:42.850984 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:42.850872 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" podUID="1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Aug 05 17:07:43.850677 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:43.850636 2573 patch_prober.go:28] interesting pod/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Aug 05 17:07:43.851254 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:43.851223 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" podUID="1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Aug 05 17:07:44.850669 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:44.850629 2573 patch_prober.go:28] interesting pod/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Aug 05 17:07:44.850886 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:44.850715 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" podUID="1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Aug 05 17:07:45.687762 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:45.687731 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:45.850623 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:45.850580 2573 patch_prober.go:28] interesting pod/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Aug 05 17:07:45.850824 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:45.850646 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" podUID="1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Aug 05 17:07:46.851078 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:46.851028 2573 patch_prober.go:28] interesting pod/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Aug 05 17:07:46.851583 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:46.851100 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" podUID="1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Aug 05 17:07:47.854680 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:47.854646 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:47.855117 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:47.854912 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:47.855683 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:47.855662 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:07:51.077216 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.077176 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:51.528295 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.528261 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:51.807115 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.807034 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75"] Aug 05 17:07:51.807398 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.807366 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57ab2728-4202-4bb4-9b3e-e79efb629a62" containerName="istio-proxy" Aug 05 17:07:51.807398 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.807394 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="57ab2728-4202-4bb4-9b3e-e79efb629a62" containerName="istio-proxy" Aug 05 17:07:51.807480 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.807408 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57ab2728-4202-4bb4-9b3e-e79efb629a62" containerName="istio-proxy" Aug 05 17:07:51.807480 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.807414 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="57ab2728-4202-4bb4-9b3e-e79efb629a62" containerName="istio-proxy" Aug 05 17:07:51.807480 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.807478 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="57ab2728-4202-4bb4-9b3e-e79efb629a62" containerName="istio-proxy" Aug 05 17:07:51.809625 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.809607 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:51.813242 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.813226 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f2130f-openshift-default-dockercfg-pd72c\"" Aug 05 17:07:51.828416 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.828369 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75"] Aug 05 17:07:51.831282 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.831251 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nqvs\" (UniqueName: \"kubernetes.io/projected/10c150af-24ee-4c4b-87fc-8107785d9d3b-kube-api-access-9nqvs\") pod \"e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:51.831414 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.831303 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-istio-envoy\") pod \"e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:51.831414 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.831330 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10c150af-24ee-4c4b-87fc-8107785d9d3b-istiod-ca-cert\") pod \"e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:51.831414 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.831409 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-credential-socket\") pod \"e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:51.831599 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.831442 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10c150af-24ee-4c4b-87fc-8107785d9d3b-istio-podinfo\") pod \"e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:51.831599 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.831487 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-workload-certs\") pod \"e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:51.831599 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.831516 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-workload-socket\") pod \"e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:51.831599 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.831574 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-istio-data\") pod \"e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:51.831817 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.831605 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10c150af-24ee-4c4b-87fc-8107785d9d3b-istio-token\") pod \"e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:51.850232 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.850201 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:51.932659 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.932626 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-credential-socket\") pod \"e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:51.932659 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.932666 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10c150af-24ee-4c4b-87fc-8107785d9d3b-istio-podinfo\") pod \"e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:51.932877 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.932802 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-workload-certs\") pod \"e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:51.932877 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.932850 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-workload-socket\") pod \"e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:51.932959 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.932912 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-istio-data\") pod \"e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:51.932959 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.932944 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10c150af-24ee-4c4b-87fc-8107785d9d3b-istio-token\") pod \"e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:51.933081 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.933017 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nqvs\" (UniqueName: \"kubernetes.io/projected/10c150af-24ee-4c4b-87fc-8107785d9d3b-kube-api-access-9nqvs\") pod \"e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:51.933081 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.933052 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-istio-envoy\") pod \"e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:51.933182 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.933080 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10c150af-24ee-4c4b-87fc-8107785d9d3b-istiod-ca-cert\") pod \"e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:51.933182 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.933113 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-workload-certs\") pod \"e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:51.933328 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.933302 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-istio-data\") pod \"e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:51.933495 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.933473 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-credential-socket\") pod \"e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:51.933597 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.933551 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-workload-socket\") pod \"e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:51.933814 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.933794 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10c150af-24ee-4c4b-87fc-8107785d9d3b-istiod-ca-cert\") pod \"e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:51.935581 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.935563 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-istio-envoy\") pod \"e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:51.935737 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.935720 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10c150af-24ee-4c4b-87fc-8107785d9d3b-istio-podinfo\") pod \"e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:51.945557 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.945530 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nqvs\" (UniqueName: \"kubernetes.io/projected/10c150af-24ee-4c4b-87fc-8107785d9d3b-kube-api-access-9nqvs\") pod \"e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:51.947858 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:51.947839 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10c150af-24ee-4c4b-87fc-8107785d9d3b-istio-token\") pod \"e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:52.002943 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:52.002916 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:52.119712 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:52.119629 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:52.258437 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:52.258412 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75"] Aug 05 17:07:52.259781 ip-10-0-132-37 kubenswrapper[2573]: W0805 17:07:52.259756 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c150af_24ee_4c4b_87fc_8107785d9d3b.slice/crio-3ac26f8ecf9d0c53b3c3f8a18bcf28ff8ded1e77d7d5718cbd1dfdeb656c387b WatchSource:0}: Error finding container 3ac26f8ecf9d0c53b3c3f8a18bcf28ff8ded1e77d7d5718cbd1dfdeb656c387b: Status 404 returned error can't find the container with id 3ac26f8ecf9d0c53b3c3f8a18bcf28ff8ded1e77d7d5718cbd1dfdeb656c387b Aug 05 17:07:52.261820 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:52.261791 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:07:52.261893 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:52.261850 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:07:52.261893 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:52.261877 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:07:52.285287 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:52.285259 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" event={"ID":"10c150af-24ee-4c4b-87fc-8107785d9d3b","Type":"ContainerStarted","Data":"3ac26f8ecf9d0c53b3c3f8a18bcf28ff8ded1e77d7d5718cbd1dfdeb656c387b"} Aug 05 17:07:53.292449 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:53.292412 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" event={"ID":"10c150af-24ee-4c4b-87fc-8107785d9d3b","Type":"ContainerStarted","Data":"a7bf1ab5e83ea0e5ec7129c77eb0c1ff592d1c5d9d98b16d5ab67d67b60be88d"} Aug 05 17:07:53.320392 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:53.320334 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" podStartSLOduration=2.320317777 podStartE2EDuration="2.320317777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:07:53.31842148 +0000 UTC m=+2788.475267382" watchObservedRunningTime="2026-08-05 17:07:53.320317777 +0000 UTC m=+2788.477163677" Aug 05 17:07:54.120442 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:54.120408 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:54.125309 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:54.125285 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:54.297198 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:54.297169 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:54.298176 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:54.298157 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:07:55.267251 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:55.267217 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:55.369612 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:55.369579 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:56.390556 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:56.390524 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:56.563206 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:56.563179 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:56.904974 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:56.904939 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:07:57.076859 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:07:57.076825 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:09.299048 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:09.299012 2573 patch_prober.go:28] interesting pod/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Aug 05 17:08:09.299522 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:09.299077 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" podUID="10c150af-24ee-4c4b-87fc-8107785d9d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Aug 05 17:08:09.371593 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:09.371556 2573 generic.go:356] "Generic (PLEG): container finished" podID="10c150af-24ee-4c4b-87fc-8107785d9d3b" containerID="a7bf1ab5e83ea0e5ec7129c77eb0c1ff592d1c5d9d98b16d5ab67d67b60be88d" exitCode=0 Aug 05 17:08:09.371765 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:09.371632 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" event={"ID":"10c150af-24ee-4c4b-87fc-8107785d9d3b","Type":"ContainerDied","Data":"a7bf1ab5e83ea0e5ec7129c77eb0c1ff592d1c5d9d98b16d5ab67d67b60be88d"} Aug 05 17:08:09.372040 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:09.372022 2573 scope.go:117] "RemoveContainer" containerID="a7bf1ab5e83ea0e5ec7129c77eb0c1ff592d1c5d9d98b16d5ab67d67b60be88d" Aug 05 17:08:09.373006 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:09.372979 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:08:09.373065 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:09.373051 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:08:09.373102 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:09.373087 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:08:10.377971 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:10.377935 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" event={"ID":"10c150af-24ee-4c4b-87fc-8107785d9d3b","Type":"ContainerStarted","Data":"d76a8bb403cc3c84659550736ea6df2bf8b9dc5b16131f830188a59365850b1c"} Aug 05 17:08:11.119960 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:11.119926 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:08:11.120309 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:11.120127 2573 patch_prober.go:28] interesting pod/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Aug 05 17:08:11.120309 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:11.120174 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" podUID="10c150af-24ee-4c4b-87fc-8107785d9d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Aug 05 17:08:12.120370 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:12.120336 2573 patch_prober.go:28] interesting pod/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Aug 05 17:08:12.120759 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:12.120410 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" podUID="10c150af-24ee-4c4b-87fc-8107785d9d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Aug 05 17:08:13.120304 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:13.120264 2573 patch_prober.go:28] interesting pod/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Aug 05 17:08:13.120534 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:13.120340 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" podUID="10c150af-24ee-4c4b-87fc-8107785d9d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Aug 05 17:08:14.120876 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:14.120836 2573 patch_prober.go:28] interesting pod/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Aug 05 17:08:14.121277 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:14.120897 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" podUID="10c150af-24ee-4c4b-87fc-8107785d9d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Aug 05 17:08:15.120448 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:15.120414 2573 patch_prober.go:28] interesting pod/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Aug 05 17:08:15.120639 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:15.120470 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" podUID="10c150af-24ee-4c4b-87fc-8107785d9d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Aug 05 17:08:15.361989 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:15.361953 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:16.124152 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:16.124119 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:08:16.124534 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:16.124510 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:08:16.125208 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:16.125191 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:08:20.199103 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:20.199070 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:38.612492 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:38.612454 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:38.727931 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:38.727891 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:38.829459 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:38.829416 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:38.910658 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:38.910570 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:39.287918 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:39.287880 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:45.512607 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:45.512578 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:45.612347 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:45.612308 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:45.748880 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:45.748848 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478"] Aug 05 17:08:45.749087 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:45.749060 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" podUID="1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a" containerName="istio-proxy" containerID="cri-o://019d0fa9cf6d356475c42a825029a80b88ef4699448b9bf783fdf72f6158f031" gracePeriod=30 Aug 05 17:08:45.944402 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:45.944293 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:46.045020 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:46.044989 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:08:47.857553 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:47.857518 2573 patch_prober.go:28] interesting pod/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 05 17:08:47.857951 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:47.857576 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" podUID="1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 05 17:08:50.990142 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:50.990114 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:08:51.090865 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.090783 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ff7p2\" (UniqueName: \"kubernetes.io/projected/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-kube-api-access-ff7p2\") pod \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " Aug 05 17:08:51.090865 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.090817 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istio-podinfo\") pod \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " Aug 05 17:08:51.090865 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.090846 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istio-data\") pod \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " Aug 05 17:08:51.091144 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.090880 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istiod-ca-cert\") pod \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " Aug 05 17:08:51.091144 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.090921 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istio-envoy\") pod \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " Aug 05 17:08:51.091144 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.090953 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-workload-certs\") pod \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " Aug 05 17:08:51.091144 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.090988 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-workload-socket\") pod \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " Aug 05 17:08:51.091144 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.091019 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-credential-socket\") pod \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " Aug 05 17:08:51.091144 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.091069 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istio-token\") pod \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\" (UID: \"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a\") " Aug 05 17:08:51.091487 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.091198 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istio-data" (OuterVolumeSpecName: "istio-data") pod "1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a" (UID: "1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:08:51.091487 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.091299 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a" (UID: "1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:08:51.091604 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.091496 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istio-data\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:08:51.091604 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.091494 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a" (UID: "1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:08:51.091604 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.091517 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-workload-certs\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:08:51.091604 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.091526 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a" (UID: "1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:08:51.091604 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.091579 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a" (UID: "1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:08:51.093033 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.093011 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a" (UID: "1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 17:08:51.093214 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.093185 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a" (UID: "1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:08:51.093479 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.093459 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-kube-api-access-ff7p2" (OuterVolumeSpecName: "kube-api-access-ff7p2") pod "1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a" (UID: "1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a"). InnerVolumeSpecName "kube-api-access-ff7p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:08:51.093538 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.093493 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istio-token" (OuterVolumeSpecName: "istio-token") pod "1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a" (UID: "1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:08:51.192623 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.192590 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istiod-ca-cert\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:08:51.192623 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.192618 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istio-envoy\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:08:51.192623 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.192629 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-workload-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:08:51.192859 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.192637 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-credential-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:08:51.192859 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.192647 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istio-token\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:08:51.192859 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.192656 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff7p2\" (UniqueName: \"kubernetes.io/projected/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-kube-api-access-ff7p2\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:08:51.192859 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.192664 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a-istio-podinfo\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:08:51.561536 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.561501 2573 generic.go:356] "Generic (PLEG): container finished" podID="1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a" containerID="019d0fa9cf6d356475c42a825029a80b88ef4699448b9bf783fdf72f6158f031" exitCode=0 Aug 05 17:08:51.561719 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.561556 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" Aug 05 17:08:51.561719 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.561566 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" event={"ID":"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a","Type":"ContainerDied","Data":"019d0fa9cf6d356475c42a825029a80b88ef4699448b9bf783fdf72f6158f031"} Aug 05 17:08:51.561719 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.561618 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478" event={"ID":"1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a","Type":"ContainerDied","Data":"df729bbdef34fd511c3f61a95c6400e4592678a871c5384719a97145b57b0296"} Aug 05 17:08:51.561719 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.561649 2573 scope.go:117] "RemoveContainer" containerID="019d0fa9cf6d356475c42a825029a80b88ef4699448b9bf783fdf72f6158f031" Aug 05 17:08:51.569992 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.569971 2573 scope.go:117] "RemoveContainer" containerID="83ff39e22a9d2f165db6c57536e8bd67413cabf65be9c3845c3520f66f254ddc" Aug 05 17:08:51.579215 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.579197 2573 scope.go:117] "RemoveContainer" containerID="019d0fa9cf6d356475c42a825029a80b88ef4699448b9bf783fdf72f6158f031" Aug 05 17:08:51.579488 ip-10-0-132-37 kubenswrapper[2573]: E0805 17:08:51.579472 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"019d0fa9cf6d356475c42a825029a80b88ef4699448b9bf783fdf72f6158f031\": container with ID starting with 019d0fa9cf6d356475c42a825029a80b88ef4699448b9bf783fdf72f6158f031 not found: ID does not exist" containerID="019d0fa9cf6d356475c42a825029a80b88ef4699448b9bf783fdf72f6158f031" Aug 05 17:08:51.579545 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.579500 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"019d0fa9cf6d356475c42a825029a80b88ef4699448b9bf783fdf72f6158f031"} err="failed to get container status \"019d0fa9cf6d356475c42a825029a80b88ef4699448b9bf783fdf72f6158f031\": rpc error: code = NotFound desc = could not find container \"019d0fa9cf6d356475c42a825029a80b88ef4699448b9bf783fdf72f6158f031\": container with ID starting with 019d0fa9cf6d356475c42a825029a80b88ef4699448b9bf783fdf72f6158f031 not found: ID does not exist" Aug 05 17:08:51.579545 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.579517 2573 scope.go:117] "RemoveContainer" containerID="83ff39e22a9d2f165db6c57536e8bd67413cabf65be9c3845c3520f66f254ddc" Aug 05 17:08:51.579731 ip-10-0-132-37 kubenswrapper[2573]: E0805 17:08:51.579713 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ff39e22a9d2f165db6c57536e8bd67413cabf65be9c3845c3520f66f254ddc\": container with ID starting with 83ff39e22a9d2f165db6c57536e8bd67413cabf65be9c3845c3520f66f254ddc not found: ID does not exist" containerID="83ff39e22a9d2f165db6c57536e8bd67413cabf65be9c3845c3520f66f254ddc" Aug 05 17:08:51.579777 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.579737 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ff39e22a9d2f165db6c57536e8bd67413cabf65be9c3845c3520f66f254ddc"} err="failed to get container status \"83ff39e22a9d2f165db6c57536e8bd67413cabf65be9c3845c3520f66f254ddc\": rpc error: code = NotFound desc = could not find container \"83ff39e22a9d2f165db6c57536e8bd67413cabf65be9c3845c3520f66f254ddc\": container with ID starting with 83ff39e22a9d2f165db6c57536e8bd67413cabf65be9c3845c3520f66f254ddc not found: ID does not exist" Aug 05 17:08:51.584627 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.584606 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478"] Aug 05 17:08:51.589472 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:51.589451 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-64d8dc-openshift-default-69588b8c4b-2l478"] Aug 05 17:08:53.304029 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:08:53.303996 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a" path="/var/lib/kubelet/pods/1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a/volumes" Aug 05 17:09:01.666779 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:01.666732 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:01.768017 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:01.767974 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:01.876783 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:01.876743 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:01.959026 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:01.958899 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:02.247425 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:02.247312 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:08.530422 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:08.530383 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:08.596892 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:08.596834 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:08.765655 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:08.765622 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75"] Aug 05 17:09:08.765835 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:08.765816 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" podUID="10c150af-24ee-4c4b-87fc-8107785d9d3b" containerName="istio-proxy" containerID="cri-o://d76a8bb403cc3c84659550736ea6df2bf8b9dc5b16131f830188a59365850b1c" gracePeriod=30 Aug 05 17:09:08.791882 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:08.791787 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:09.046368 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.046265 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:09.328671 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.328584 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:09.595778 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.595694 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw"] Aug 05 17:09:09.596150 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.596018 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a" containerName="istio-proxy" Aug 05 17:09:09.596150 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.596030 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a" containerName="istio-proxy" Aug 05 17:09:09.596150 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.596047 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a" containerName="istio-proxy" Aug 05 17:09:09.596150 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.596052 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a" containerName="istio-proxy" Aug 05 17:09:09.596150 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.596103 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a" containerName="istio-proxy" Aug 05 17:09:09.596150 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.596112 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="57ab2728-4202-4bb4-9b3e-e79efb629a62" containerName="istio-proxy" Aug 05 17:09:09.596150 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.596117 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e0415c6-1bd9-4cf1-9755-9c6b0e8cec9a" containerName="istio-proxy" Aug 05 17:09:09.598053 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.598035 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:09.603303 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.603282 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-02b433-openshift-default-dockercfg-nr8sf\"" Aug 05 17:09:09.615352 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.615330 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw"] Aug 05 17:09:09.649451 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.649403 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-workload-certs\") pod \"e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:09.649631 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.649471 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-credential-socket\") pod \"e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:09.649631 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.649498 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istio-data\") pod \"e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:09.649631 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.649514 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istio-podinfo\") pod \"e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:09.649631 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.649544 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkzcg\" (UniqueName: \"kubernetes.io/projected/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-kube-api-access-kkzcg\") pod \"e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:09.649787 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.649668 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istio-token\") pod \"e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:09.649787 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.649706 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istiod-ca-cert\") pod \"e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:09.649787 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.649768 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-workload-socket\") pod \"e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:09.649900 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.649799 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istio-envoy\") pod \"e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:09.651951 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.651924 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:09.750335 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.750295 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istio-envoy\") pod \"e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:09.750557 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.750366 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-workload-certs\") pod \"e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:09.750557 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.750418 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-credential-socket\") pod \"e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:09.750688 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.750608 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istio-data\") pod \"e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:09.750688 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.750636 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istio-podinfo\") pod \"e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:09.750688 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.750676 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkzcg\" (UniqueName: \"kubernetes.io/projected/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-kube-api-access-kkzcg\") pod \"e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:09.750831 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.750724 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istio-token\") pod \"e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:09.751224 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.751189 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-credential-socket\") pod \"e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:09.751347 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.751256 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-workload-certs\") pod \"e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:09.751347 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.751287 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istio-data\") pod \"e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:09.752045 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.751564 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istiod-ca-cert\") pod \"e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:09.752045 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.751631 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-workload-socket\") pod \"e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:09.752045 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.751989 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-workload-socket\") pod \"e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:09.758364 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.752474 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istiod-ca-cert\") pod \"e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:09.760319 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.760257 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istio-envoy\") pod \"e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:09.760824 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.760799 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istio-podinfo\") pod \"e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:09.761350 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.761319 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istio-token\") pod \"e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:09.761858 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.761838 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkzcg\" (UniqueName: \"kubernetes.io/projected/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-kube-api-access-kkzcg\") pod \"e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:09.907938 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:09.907851 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:10.050988 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:10.050953 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw"] Aug 05 17:09:10.053817 ip-10-0-132-37 kubenswrapper[2573]: W0805 17:09:10.053781 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod265e37c0_bc0b_421c_bbfa_2e7c74d7d731.slice/crio-f77385288ab73e204bcc87446bb23af24d489d499eedefeba7f3249d2fbcb932 WatchSource:0}: Error finding container f77385288ab73e204bcc87446bb23af24d489d499eedefeba7f3249d2fbcb932: Status 404 returned error can't find the container with id f77385288ab73e204bcc87446bb23af24d489d499eedefeba7f3249d2fbcb932 Aug 05 17:09:10.056054 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:10.056015 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:09:10.056151 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:10.056091 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:09:10.056151 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:10.056122 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:09:10.556233 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:10.556183 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:10.654398 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:10.654345 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" event={"ID":"265e37c0-bc0b-421c-bbfa-2e7c74d7d731","Type":"ContainerStarted","Data":"85caf50cac15164e880f6bf4e4e3999d66d07a79891f31e205fde121c9023834"} Aug 05 17:09:10.654794 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:10.654409 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" event={"ID":"265e37c0-bc0b-421c-bbfa-2e7c74d7d731","Type":"ContainerStarted","Data":"f77385288ab73e204bcc87446bb23af24d489d499eedefeba7f3249d2fbcb932"} Aug 05 17:09:10.677722 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:10.677672 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" podStartSLOduration=1.677655536 podStartE2EDuration="1.677655536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:09:10.67579208 +0000 UTC m=+2865.832637982" watchObservedRunningTime="2026-08-05 17:09:10.677655536 +0000 UTC m=+2865.834501488" Aug 05 17:09:10.908176 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:10.908093 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:11.913876 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:11.913841 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:12.594303 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:12.594249 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:12.663854 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:12.663826 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:12.664720 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:12.664698 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:12.695025 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:12.694982 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:13.819994 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:13.819966 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:14.006838 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.006816 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:09:14.088866 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.088782 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-istio-data\") pod \"10c150af-24ee-4c4b-87fc-8107785d9d3b\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " Aug 05 17:09:14.088866 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.088823 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-istio-envoy\") pod \"10c150af-24ee-4c4b-87fc-8107785d9d3b\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " Aug 05 17:09:14.088866 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.088868 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nqvs\" (UniqueName: \"kubernetes.io/projected/10c150af-24ee-4c4b-87fc-8107785d9d3b-kube-api-access-9nqvs\") pod \"10c150af-24ee-4c4b-87fc-8107785d9d3b\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " Aug 05 17:09:14.089159 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.088893 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-workload-certs\") pod \"10c150af-24ee-4c4b-87fc-8107785d9d3b\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " Aug 05 17:09:14.089159 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.088909 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-workload-socket\") pod \"10c150af-24ee-4c4b-87fc-8107785d9d3b\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " Aug 05 17:09:14.089159 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.088939 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10c150af-24ee-4c4b-87fc-8107785d9d3b-istio-token\") pod \"10c150af-24ee-4c4b-87fc-8107785d9d3b\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " Aug 05 17:09:14.089159 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.088978 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-credential-socket\") pod \"10c150af-24ee-4c4b-87fc-8107785d9d3b\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " Aug 05 17:09:14.089159 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.089027 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10c150af-24ee-4c4b-87fc-8107785d9d3b-istio-podinfo\") pod \"10c150af-24ee-4c4b-87fc-8107785d9d3b\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " Aug 05 17:09:14.089159 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.089067 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10c150af-24ee-4c4b-87fc-8107785d9d3b-istiod-ca-cert\") pod \"10c150af-24ee-4c4b-87fc-8107785d9d3b\" (UID: \"10c150af-24ee-4c4b-87fc-8107785d9d3b\") " Aug 05 17:09:14.089159 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.089105 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-istio-data" (OuterVolumeSpecName: "istio-data") pod "10c150af-24ee-4c4b-87fc-8107785d9d3b" (UID: "10c150af-24ee-4c4b-87fc-8107785d9d3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:09:14.089506 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.089264 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "10c150af-24ee-4c4b-87fc-8107785d9d3b" (UID: "10c150af-24ee-4c4b-87fc-8107785d9d3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:09:14.089506 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.089290 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "10c150af-24ee-4c4b-87fc-8107785d9d3b" (UID: "10c150af-24ee-4c4b-87fc-8107785d9d3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:09:14.089506 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.089424 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-workload-certs\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:09:14.089506 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.089449 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-workload-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:09:14.089506 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.089463 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-istio-data\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:09:14.089757 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.089545 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "10c150af-24ee-4c4b-87fc-8107785d9d3b" (UID: "10c150af-24ee-4c4b-87fc-8107785d9d3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:09:14.089816 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.089761 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10c150af-24ee-4c4b-87fc-8107785d9d3b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "10c150af-24ee-4c4b-87fc-8107785d9d3b" (UID: "10c150af-24ee-4c4b-87fc-8107785d9d3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:09:14.091438 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.091407 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c150af-24ee-4c4b-87fc-8107785d9d3b-istio-token" (OuterVolumeSpecName: "istio-token") pod "10c150af-24ee-4c4b-87fc-8107785d9d3b" (UID: "10c150af-24ee-4c4b-87fc-8107785d9d3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:09:14.091694 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.091676 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c150af-24ee-4c4b-87fc-8107785d9d3b-kube-api-access-9nqvs" (OuterVolumeSpecName: "kube-api-access-9nqvs") pod "10c150af-24ee-4c4b-87fc-8107785d9d3b" (UID: "10c150af-24ee-4c4b-87fc-8107785d9d3b"). InnerVolumeSpecName "kube-api-access-9nqvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:09:14.091803 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.091783 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "10c150af-24ee-4c4b-87fc-8107785d9d3b" (UID: "10c150af-24ee-4c4b-87fc-8107785d9d3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:09:14.091857 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.091788 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10c150af-24ee-4c4b-87fc-8107785d9d3b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "10c150af-24ee-4c4b-87fc-8107785d9d3b" (UID: "10c150af-24ee-4c4b-87fc-8107785d9d3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 17:09:14.196174 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.196131 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-credential-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:09:14.196398 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.196181 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10c150af-24ee-4c4b-87fc-8107785d9d3b-istio-podinfo\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:09:14.196398 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.196366 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10c150af-24ee-4c4b-87fc-8107785d9d3b-istiod-ca-cert\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:09:14.196515 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.196412 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10c150af-24ee-4c4b-87fc-8107785d9d3b-istio-envoy\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:09:14.196515 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.196426 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nqvs\" (UniqueName: \"kubernetes.io/projected/10c150af-24ee-4c4b-87fc-8107785d9d3b-kube-api-access-9nqvs\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:09:14.196515 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.196442 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10c150af-24ee-4c4b-87fc-8107785d9d3b-istio-token\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:09:14.281167 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.281131 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:14.582261 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.582227 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:14.674903 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.674864 2573 generic.go:356] "Generic (PLEG): container finished" podID="10c150af-24ee-4c4b-87fc-8107785d9d3b" containerID="d76a8bb403cc3c84659550736ea6df2bf8b9dc5b16131f830188a59365850b1c" exitCode=0 Aug 05 17:09:14.675074 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.674936 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" Aug 05 17:09:14.675074 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.674954 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" event={"ID":"10c150af-24ee-4c4b-87fc-8107785d9d3b","Type":"ContainerDied","Data":"d76a8bb403cc3c84659550736ea6df2bf8b9dc5b16131f830188a59365850b1c"} Aug 05 17:09:14.675074 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.675009 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75" event={"ID":"10c150af-24ee-4c4b-87fc-8107785d9d3b","Type":"ContainerDied","Data":"3ac26f8ecf9d0c53b3c3f8a18bcf28ff8ded1e77d7d5718cbd1dfdeb656c387b"} Aug 05 17:09:14.675074 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.675032 2573 scope.go:117] "RemoveContainer" containerID="d76a8bb403cc3c84659550736ea6df2bf8b9dc5b16131f830188a59365850b1c" Aug 05 17:09:14.680002 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.679979 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:14.684634 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.684619 2573 scope.go:117] "RemoveContainer" containerID="a7bf1ab5e83ea0e5ec7129c77eb0c1ff592d1c5d9d98b16d5ab67d67b60be88d" Aug 05 17:09:14.692142 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.692124 2573 scope.go:117] "RemoveContainer" containerID="d76a8bb403cc3c84659550736ea6df2bf8b9dc5b16131f830188a59365850b1c" Aug 05 17:09:14.692394 ip-10-0-132-37 kubenswrapper[2573]: E0805 17:09:14.692355 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d76a8bb403cc3c84659550736ea6df2bf8b9dc5b16131f830188a59365850b1c\": container with ID starting with d76a8bb403cc3c84659550736ea6df2bf8b9dc5b16131f830188a59365850b1c not found: ID does not exist" containerID="d76a8bb403cc3c84659550736ea6df2bf8b9dc5b16131f830188a59365850b1c" Aug 05 17:09:14.692453 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.692404 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d76a8bb403cc3c84659550736ea6df2bf8b9dc5b16131f830188a59365850b1c"} err="failed to get container status \"d76a8bb403cc3c84659550736ea6df2bf8b9dc5b16131f830188a59365850b1c\": rpc error: code = NotFound desc = could not find container \"d76a8bb403cc3c84659550736ea6df2bf8b9dc5b16131f830188a59365850b1c\": container with ID starting with d76a8bb403cc3c84659550736ea6df2bf8b9dc5b16131f830188a59365850b1c not found: ID does not exist" Aug 05 17:09:14.692453 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.692421 2573 scope.go:117] "RemoveContainer" containerID="a7bf1ab5e83ea0e5ec7129c77eb0c1ff592d1c5d9d98b16d5ab67d67b60be88d" Aug 05 17:09:14.692629 ip-10-0-132-37 kubenswrapper[2573]: E0805 17:09:14.692615 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7bf1ab5e83ea0e5ec7129c77eb0c1ff592d1c5d9d98b16d5ab67d67b60be88d\": container with ID starting with a7bf1ab5e83ea0e5ec7129c77eb0c1ff592d1c5d9d98b16d5ab67d67b60be88d not found: ID does not exist" containerID="a7bf1ab5e83ea0e5ec7129c77eb0c1ff592d1c5d9d98b16d5ab67d67b60be88d" Aug 05 17:09:14.692670 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.692632 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7bf1ab5e83ea0e5ec7129c77eb0c1ff592d1c5d9d98b16d5ab67d67b60be88d"} err="failed to get container status \"a7bf1ab5e83ea0e5ec7129c77eb0c1ff592d1c5d9d98b16d5ab67d67b60be88d\": rpc error: code = NotFound desc = could not find container \"a7bf1ab5e83ea0e5ec7129c77eb0c1ff592d1c5d9d98b16d5ab67d67b60be88d\": container with ID starting with a7bf1ab5e83ea0e5ec7129c77eb0c1ff592d1c5d9d98b16d5ab67d67b60be88d not found: ID does not exist" Aug 05 17:09:14.704766 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.704742 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75"] Aug 05 17:09:14.714339 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:14.714319 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f2130f-openshift-default-6cc76bddf8-4wd75"] Aug 05 17:09:15.304392 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:15.304330 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10c150af-24ee-4c4b-87fc-8107785d9d3b" path="/var/lib/kubelet/pods/10c150af-24ee-4c4b-87fc-8107785d9d3b/volumes" Aug 05 17:09:26.740682 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:26.740645 2573 generic.go:356] "Generic (PLEG): container finished" podID="265e37c0-bc0b-421c-bbfa-2e7c74d7d731" containerID="85caf50cac15164e880f6bf4e4e3999d66d07a79891f31e205fde121c9023834" exitCode=0 Aug 05 17:09:26.741137 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:26.740720 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" event={"ID":"265e37c0-bc0b-421c-bbfa-2e7c74d7d731","Type":"ContainerDied","Data":"85caf50cac15164e880f6bf4e4e3999d66d07a79891f31e205fde121c9023834"} Aug 05 17:09:26.741193 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:26.741140 2573 scope.go:117] "RemoveContainer" containerID="85caf50cac15164e880f6bf4e4e3999d66d07a79891f31e205fde121c9023834" Aug 05 17:09:26.742092 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:26.742066 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:09:26.742151 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:26.742136 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:09:26.742206 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:26.742194 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:09:27.664813 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:27.664773 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:27.750064 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:27.750018 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" event={"ID":"265e37c0-bc0b-421c-bbfa-2e7c74d7d731","Type":"ContainerStarted","Data":"3d0926befb0a8da1e78f1bf30e73f94ffe5e625cfee0b1e66c97ad64654c49da"} Aug 05 17:09:27.908643 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:27.908607 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:27.909628 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:27.909601 2573 patch_prober.go:28] interesting pod/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Aug 05 17:09:27.909735 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:27.909651 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" podUID="265e37c0-bc0b-421c-bbfa-2e7c74d7d731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Aug 05 17:09:28.908689 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:28.908652 2573 patch_prober.go:28] interesting pod/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Aug 05 17:09:28.909095 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:28.908709 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" podUID="265e37c0-bc0b-421c-bbfa-2e7c74d7d731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Aug 05 17:09:29.908350 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:29.908314 2573 patch_prober.go:28] interesting pod/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Aug 05 17:09:29.908560 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:29.908393 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" podUID="265e37c0-bc0b-421c-bbfa-2e7c74d7d731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Aug 05 17:09:30.908938 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:30.908903 2573 patch_prober.go:28] interesting pod/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Aug 05 17:09:30.909342 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:30.908955 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" podUID="265e37c0-bc0b-421c-bbfa-2e7c74d7d731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Aug 05 17:09:31.908708 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:31.908659 2573 patch_prober.go:28] interesting pod/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Aug 05 17:09:31.908923 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:31.908726 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" podUID="265e37c0-bc0b-421c-bbfa-2e7c74d7d731" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Aug 05 17:09:32.912045 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:32.912002 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:32.912524 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:32.912294 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:32.912935 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:32.912920 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:09:37.534689 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:37.534655 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:37.866658 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:37.866561 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:38.135549 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.135453 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp"] Aug 05 17:09:38.135945 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.135918 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10c150af-24ee-4c4b-87fc-8107785d9d3b" containerName="istio-proxy" Aug 05 17:09:38.135945 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.135937 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="10c150af-24ee-4c4b-87fc-8107785d9d3b" containerName="istio-proxy" Aug 05 17:09:38.136066 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.135947 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10c150af-24ee-4c4b-87fc-8107785d9d3b" containerName="istio-proxy" Aug 05 17:09:38.136066 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.135953 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="10c150af-24ee-4c4b-87fc-8107785d9d3b" containerName="istio-proxy" Aug 05 17:09:38.136066 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.136012 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="10c150af-24ee-4c4b-87fc-8107785d9d3b" containerName="istio-proxy" Aug 05 17:09:38.136066 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.136022 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="10c150af-24ee-4c4b-87fc-8107785d9d3b" containerName="istio-proxy" Aug 05 17:09:38.139021 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.138998 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.142384 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.142353 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-46e6bf-openshift-default-dockercfg-9t5x5\"" Aug 05 17:09:38.158144 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.158116 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp"] Aug 05 17:09:38.184447 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.184420 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:38.208533 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.208504 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec6581e2-a031-44ce-860f-de3dd9c45642-istiod-ca-cert\") pod \"e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.208685 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.208543 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwbzv\" (UniqueName: \"kubernetes.io/projected/ec6581e2-a031-44ce-860f-de3dd9c45642-kube-api-access-nwbzv\") pod \"e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.208685 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.208577 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec6581e2-a031-44ce-860f-de3dd9c45642-istio-podinfo\") pod \"e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.208685 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.208602 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec6581e2-a031-44ce-860f-de3dd9c45642-istio-token\") pod \"e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.208685 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.208637 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-istio-envoy\") pod \"e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.208911 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.208719 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-credential-socket\") pod \"e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.208911 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.208746 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-istio-data\") pod \"e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.208911 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.208771 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-workload-certs\") pod \"e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.208911 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.208834 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-workload-socket\") pod \"e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.309254 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.309217 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwbzv\" (UniqueName: \"kubernetes.io/projected/ec6581e2-a031-44ce-860f-de3dd9c45642-kube-api-access-nwbzv\") pod \"e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.309254 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.309262 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec6581e2-a031-44ce-860f-de3dd9c45642-istio-podinfo\") pod \"e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.309557 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.309277 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec6581e2-a031-44ce-860f-de3dd9c45642-istio-token\") pod \"e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.309557 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.309500 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-istio-envoy\") pod \"e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.309677 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.309562 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-credential-socket\") pod \"e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.309677 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.309589 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-istio-data\") pod \"e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.309677 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.309620 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-workload-certs\") pod \"e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.309828 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.309685 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-workload-socket\") pod \"e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.309828 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.309720 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec6581e2-a031-44ce-860f-de3dd9c45642-istiod-ca-cert\") pod \"e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.310020 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.309993 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-credential-socket\") pod \"e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.310121 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.310028 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-istio-data\") pod \"e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.310121 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.310047 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-workload-socket\") pod \"e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.310207 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.310172 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-workload-certs\") pod \"e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.310528 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.310507 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec6581e2-a031-44ce-860f-de3dd9c45642-istiod-ca-cert\") pod \"e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.311757 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.311739 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-istio-envoy\") pod \"e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.311968 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.311951 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec6581e2-a031-44ce-860f-de3dd9c45642-istio-podinfo\") pod \"e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.319215 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.319182 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec6581e2-a031-44ce-860f-de3dd9c45642-istio-token\") pod \"e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.319910 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.319892 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwbzv\" (UniqueName: \"kubernetes.io/projected/ec6581e2-a031-44ce-860f-de3dd9c45642-kube-api-access-nwbzv\") pod \"e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.356056 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.356027 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:38.449477 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.449367 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:38.592321 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.592293 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp"] Aug 05 17:09:38.594034 ip-10-0-132-37 kubenswrapper[2573]: W0805 17:09:38.594008 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec6581e2_a031_44ce_860f_de3dd9c45642.slice/crio-63cf5dbdf8dcaa8647b487a10ce9a03097e71a66145cfb431f1987d166e69069 WatchSource:0}: Error finding container 63cf5dbdf8dcaa8647b487a10ce9a03097e71a66145cfb431f1987d166e69069: Status 404 returned error can't find the container with id 63cf5dbdf8dcaa8647b487a10ce9a03097e71a66145cfb431f1987d166e69069 Aug 05 17:09:38.596245 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.596219 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:09:38.596306 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.596278 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:09:38.596342 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.596308 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:09:38.804389 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.804337 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" event={"ID":"ec6581e2-a031-44ce-860f-de3dd9c45642","Type":"ContainerStarted","Data":"10e6e9621112a862b37303989f0ceea93dd4e51ce249dbff3d7aa172fe9f5774"} Aug 05 17:09:38.804559 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.804420 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" event={"ID":"ec6581e2-a031-44ce-860f-de3dd9c45642","Type":"ContainerStarted","Data":"63cf5dbdf8dcaa8647b487a10ce9a03097e71a66145cfb431f1987d166e69069"} Aug 05 17:09:38.833705 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:38.833649 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" podStartSLOduration=0.833632776 podStartE2EDuration="833.632776ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:09:38.831328606 +0000 UTC m=+2893.988174509" watchObservedRunningTime="2026-08-05 17:09:38.833632776 +0000 UTC m=+2893.990478677" Aug 05 17:09:39.450272 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:39.450231 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:39.455696 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:39.455668 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:39.810011 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:39.809973 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:39.810921 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:39.810903 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:40.844059 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:40.844015 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:40.968093 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:40.968061 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:42.047654 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:42.047620 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:42.389463 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:42.389356 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:42.672646 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:42.672560 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:42.767144 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:42.767102 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:09:54.811770 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:54.811735 2573 patch_prober.go:28] interesting pod/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Aug 05 17:09:54.812193 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:54.811828 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" podUID="ec6581e2-a031-44ce-860f-de3dd9c45642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Aug 05 17:09:54.884478 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:54.884442 2573 generic.go:356] "Generic (PLEG): container finished" podID="ec6581e2-a031-44ce-860f-de3dd9c45642" containerID="10e6e9621112a862b37303989f0ceea93dd4e51ce249dbff3d7aa172fe9f5774" exitCode=0 Aug 05 17:09:54.884667 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:54.884507 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" event={"ID":"ec6581e2-a031-44ce-860f-de3dd9c45642","Type":"ContainerDied","Data":"10e6e9621112a862b37303989f0ceea93dd4e51ce249dbff3d7aa172fe9f5774"} Aug 05 17:09:54.884929 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:54.884911 2573 scope.go:117] "RemoveContainer" containerID="10e6e9621112a862b37303989f0ceea93dd4e51ce249dbff3d7aa172fe9f5774" Aug 05 17:09:54.885962 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:54.885931 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:09:54.886047 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:54.886003 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:09:54.886047 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:54.886033 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:09:55.890541 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:55.890504 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" event={"ID":"ec6581e2-a031-44ce-860f-de3dd9c45642","Type":"ContainerStarted","Data":"865cf0e0ff280ebc3aa84057834b58ceef0cedc88fe538e8b7f535c4389d8004"} Aug 05 17:09:56.450512 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:56.450471 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:09:56.450966 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:56.450832 2573 patch_prober.go:28] interesting pod/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Aug 05 17:09:56.451161 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:56.451136 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" podUID="ec6581e2-a031-44ce-860f-de3dd9c45642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Aug 05 17:09:57.450014 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:57.449974 2573 patch_prober.go:28] interesting pod/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Aug 05 17:09:57.450682 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:57.450644 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" podUID="ec6581e2-a031-44ce-860f-de3dd9c45642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Aug 05 17:09:58.451802 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:58.451765 2573 patch_prober.go:28] interesting pod/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Aug 05 17:09:58.452311 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:58.451835 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" podUID="ec6581e2-a031-44ce-860f-de3dd9c45642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Aug 05 17:09:59.450482 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:59.450441 2573 patch_prober.go:28] interesting pod/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Aug 05 17:09:59.450786 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:09:59.450757 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" podUID="ec6581e2-a031-44ce-860f-de3dd9c45642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Aug 05 17:10:00.450853 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:00.450802 2573 patch_prober.go:28] interesting pod/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Aug 05 17:10:00.451400 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:00.450874 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" podUID="ec6581e2-a031-44ce-860f-de3dd9c45642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Aug 05 17:10:01.454451 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:01.454422 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:10:01.454925 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:01.454687 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:10:01.455427 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:01.455410 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:10:05.753542 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:05.753510 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:10:06.413200 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:06.413152 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:10:06.529726 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:06.529668 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:10:09.706137 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:09.706102 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:10:38.910592 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:38.910557 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm"] Aug 05 17:10:38.913194 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:38.913179 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" Aug 05 17:10:38.917410 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:38.917368 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-46e6bf\"/\"openshift-service-ca.crt\"" Aug 05 17:10:38.917567 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:38.917439 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-46e6bf\"/\"kube-root-ca.crt\"" Aug 05 17:10:38.917567 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:38.917478 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-46e6bf\"/\"default-dockercfg-c2gzl\"" Aug 05 17:10:38.917567 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:38.917515 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-46e6bf\"/\"auth-test-model-46e6bf-kserve-self-signed-certs\"" Aug 05 17:10:38.926593 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:38.926571 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm"] Aug 05 17:10:38.937575 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:38.937546 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:10:38.963037 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:38.963009 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc55f55-7f4f-440e-be40-5255dd100699-model-cache\") pod \"auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm\" (UID: \"8fc55f55-7f4f-440e-be40-5255dd100699\") " pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" Aug 05 17:10:38.963185 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:38.963039 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc55f55-7f4f-440e-be40-5255dd100699-tmp-dir\") pod \"auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm\" (UID: \"8fc55f55-7f4f-440e-be40-5255dd100699\") " pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" Aug 05 17:10:38.963185 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:38.963077 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc55f55-7f4f-440e-be40-5255dd100699-home\") pod \"auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm\" (UID: \"8fc55f55-7f4f-440e-be40-5255dd100699\") " pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" Aug 05 17:10:38.963185 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:38.963174 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc55f55-7f4f-440e-be40-5255dd100699-dshm\") pod \"auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm\" (UID: \"8fc55f55-7f4f-440e-be40-5255dd100699\") " pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" Aug 05 17:10:38.963342 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:38.963211 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s22h8\" (UniqueName: \"kubernetes.io/projected/8fc55f55-7f4f-440e-be40-5255dd100699-kube-api-access-s22h8\") pod \"auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm\" (UID: \"8fc55f55-7f4f-440e-be40-5255dd100699\") " pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" Aug 05 17:10:38.963342 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:38.963297 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc55f55-7f4f-440e-be40-5255dd100699-tls-certs\") pod \"auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm\" (UID: \"8fc55f55-7f4f-440e-be40-5255dd100699\") " pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" Aug 05 17:10:39.064489 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:39.064453 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc55f55-7f4f-440e-be40-5255dd100699-home\") pod \"auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm\" (UID: \"8fc55f55-7f4f-440e-be40-5255dd100699\") " pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" Aug 05 17:10:39.064691 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:39.064500 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc55f55-7f4f-440e-be40-5255dd100699-dshm\") pod \"auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm\" (UID: \"8fc55f55-7f4f-440e-be40-5255dd100699\") " pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" Aug 05 17:10:39.064691 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:39.064526 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s22h8\" (UniqueName: \"kubernetes.io/projected/8fc55f55-7f4f-440e-be40-5255dd100699-kube-api-access-s22h8\") pod \"auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm\" (UID: \"8fc55f55-7f4f-440e-be40-5255dd100699\") " pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" Aug 05 17:10:39.064691 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:39.064569 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc55f55-7f4f-440e-be40-5255dd100699-tls-certs\") pod \"auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm\" (UID: \"8fc55f55-7f4f-440e-be40-5255dd100699\") " pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" Aug 05 17:10:39.064691 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:39.064636 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc55f55-7f4f-440e-be40-5255dd100699-model-cache\") pod \"auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm\" (UID: \"8fc55f55-7f4f-440e-be40-5255dd100699\") " pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" Aug 05 17:10:39.064691 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:39.064677 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc55f55-7f4f-440e-be40-5255dd100699-tmp-dir\") pod \"auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm\" (UID: \"8fc55f55-7f4f-440e-be40-5255dd100699\") " pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" Aug 05 17:10:39.064994 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:39.064966 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc55f55-7f4f-440e-be40-5255dd100699-home\") pod \"auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm\" (UID: \"8fc55f55-7f4f-440e-be40-5255dd100699\") " pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" Aug 05 17:10:39.065204 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:39.065184 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc55f55-7f4f-440e-be40-5255dd100699-tmp-dir\") pod \"auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm\" (UID: \"8fc55f55-7f4f-440e-be40-5255dd100699\") " pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" Aug 05 17:10:39.065485 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:39.065461 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc55f55-7f4f-440e-be40-5255dd100699-model-cache\") pod \"auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm\" (UID: \"8fc55f55-7f4f-440e-be40-5255dd100699\") " pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" Aug 05 17:10:39.067701 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:39.067676 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc55f55-7f4f-440e-be40-5255dd100699-tls-certs\") pod \"auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm\" (UID: \"8fc55f55-7f4f-440e-be40-5255dd100699\") " pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" Aug 05 17:10:39.069140 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:39.069114 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc55f55-7f4f-440e-be40-5255dd100699-dshm\") pod \"auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm\" (UID: \"8fc55f55-7f4f-440e-be40-5255dd100699\") " pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" Aug 05 17:10:39.075210 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:39.074891 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s22h8\" (UniqueName: \"kubernetes.io/projected/8fc55f55-7f4f-440e-be40-5255dd100699-kube-api-access-s22h8\") pod \"auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm\" (UID: \"8fc55f55-7f4f-440e-be40-5255dd100699\") " pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" Aug 05 17:10:39.223437 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:39.223326 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" Aug 05 17:10:39.376715 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:39.376674 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm"] Aug 05 17:10:39.379871 ip-10-0-132-37 kubenswrapper[2573]: W0805 17:10:39.379837 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fc55f55_7f4f_440e_be40_5255dd100699.slice/crio-9d56e1d6f66b774ed04489309dfc309113f18401b9d77ac980725e8c0bd8e1ee WatchSource:0}: Error finding container 9d56e1d6f66b774ed04489309dfc309113f18401b9d77ac980725e8c0bd8e1ee: Status 404 returned error can't find the container with id 9d56e1d6f66b774ed04489309dfc309113f18401b9d77ac980725e8c0bd8e1ee Aug 05 17:10:40.086419 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:40.086365 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" event={"ID":"8fc55f55-7f4f-440e-be40-5255dd100699","Type":"ContainerStarted","Data":"3e7e2945088c65e38e3ebd0f6a5e39f434f5875a8d7edf68225722b0f5080b29"} Aug 05 17:10:40.086419 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:40.086423 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" event={"ID":"8fc55f55-7f4f-440e-be40-5255dd100699","Type":"ContainerStarted","Data":"9d56e1d6f66b774ed04489309dfc309113f18401b9d77ac980725e8c0bd8e1ee"} Aug 05 17:10:40.109227 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:40.109179 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" podStartSLOduration=2.109164325 podStartE2EDuration="2.109164325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:10:40.108549893 +0000 UTC m=+2955.265395794" watchObservedRunningTime="2026-08-05 17:10:40.109164325 +0000 UTC m=+2955.266010226" Aug 05 17:10:49.223512 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:49.223419 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" Aug 05 17:10:49.223512 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:49.223467 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" Aug 05 17:10:49.235990 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:49.235961 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" Aug 05 17:10:50.144007 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:10:50.143973 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" Aug 05 17:11:43.771203 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:11:43.771163 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:11:51.499856 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:11:51.499816 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:13:33.731001 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:33.730942 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:13:38.078903 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:38.078865 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:13:39.021430 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:39.021398 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm"] Aug 05 17:13:39.041790 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:39.041763 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:13:44.502764 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:44.502728 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:13:54.040776 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:54.040731 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" podUID="8fc55f55-7f4f-440e-be40-5255dd100699" containerName="main" containerID="cri-o://3e7e2945088c65e38e3ebd0f6a5e39f434f5875a8d7edf68225722b0f5080b29" gracePeriod=45 Aug 05 17:13:54.282028 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:54.282000 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" Aug 05 17:13:54.331028 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:54.330947 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc55f55-7f4f-440e-be40-5255dd100699-model-cache\") pod \"8fc55f55-7f4f-440e-be40-5255dd100699\" (UID: \"8fc55f55-7f4f-440e-be40-5255dd100699\") " Aug 05 17:13:54.331028 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:54.330987 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc55f55-7f4f-440e-be40-5255dd100699-tls-certs\") pod \"8fc55f55-7f4f-440e-be40-5255dd100699\" (UID: \"8fc55f55-7f4f-440e-be40-5255dd100699\") " Aug 05 17:13:54.331028 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:54.331016 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc55f55-7f4f-440e-be40-5255dd100699-dshm\") pod \"8fc55f55-7f4f-440e-be40-5255dd100699\" (UID: \"8fc55f55-7f4f-440e-be40-5255dd100699\") " Aug 05 17:13:54.331302 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:54.331058 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s22h8\" (UniqueName: \"kubernetes.io/projected/8fc55f55-7f4f-440e-be40-5255dd100699-kube-api-access-s22h8\") pod \"8fc55f55-7f4f-440e-be40-5255dd100699\" (UID: \"8fc55f55-7f4f-440e-be40-5255dd100699\") " Aug 05 17:13:54.331302 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:54.331083 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc55f55-7f4f-440e-be40-5255dd100699-tmp-dir\") pod \"8fc55f55-7f4f-440e-be40-5255dd100699\" (UID: \"8fc55f55-7f4f-440e-be40-5255dd100699\") " Aug 05 17:13:54.331302 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:54.331136 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc55f55-7f4f-440e-be40-5255dd100699-home\") pod \"8fc55f55-7f4f-440e-be40-5255dd100699\" (UID: \"8fc55f55-7f4f-440e-be40-5255dd100699\") " Aug 05 17:13:54.331302 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:54.331258 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc55f55-7f4f-440e-be40-5255dd100699-model-cache" (OuterVolumeSpecName: "model-cache") pod "8fc55f55-7f4f-440e-be40-5255dd100699" (UID: "8fc55f55-7f4f-440e-be40-5255dd100699"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:13:54.331517 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:54.331417 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc55f55-7f4f-440e-be40-5255dd100699-model-cache\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:13:54.331517 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:54.331427 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc55f55-7f4f-440e-be40-5255dd100699-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8fc55f55-7f4f-440e-be40-5255dd100699" (UID: "8fc55f55-7f4f-440e-be40-5255dd100699"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:13:54.331517 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:54.331474 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc55f55-7f4f-440e-be40-5255dd100699-home" (OuterVolumeSpecName: "home") pod "8fc55f55-7f4f-440e-be40-5255dd100699" (UID: "8fc55f55-7f4f-440e-be40-5255dd100699"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:13:54.333356 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:54.333334 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fc55f55-7f4f-440e-be40-5255dd100699-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8fc55f55-7f4f-440e-be40-5255dd100699" (UID: "8fc55f55-7f4f-440e-be40-5255dd100699"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:13:54.333356 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:54.333338 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc55f55-7f4f-440e-be40-5255dd100699-dshm" (OuterVolumeSpecName: "dshm") pod "8fc55f55-7f4f-440e-be40-5255dd100699" (UID: "8fc55f55-7f4f-440e-be40-5255dd100699"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:13:54.333531 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:54.333407 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc55f55-7f4f-440e-be40-5255dd100699-kube-api-access-s22h8" (OuterVolumeSpecName: "kube-api-access-s22h8") pod "8fc55f55-7f4f-440e-be40-5255dd100699" (UID: "8fc55f55-7f4f-440e-be40-5255dd100699"). InnerVolumeSpecName "kube-api-access-s22h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:13:54.432120 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:54.432088 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s22h8\" (UniqueName: \"kubernetes.io/projected/8fc55f55-7f4f-440e-be40-5255dd100699-kube-api-access-s22h8\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:13:54.432120 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:54.432118 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc55f55-7f4f-440e-be40-5255dd100699-tmp-dir\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:13:54.432340 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:54.432132 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc55f55-7f4f-440e-be40-5255dd100699-home\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:13:54.432340 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:54.432144 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc55f55-7f4f-440e-be40-5255dd100699-tls-certs\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:13:54.432340 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:54.432155 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc55f55-7f4f-440e-be40-5255dd100699-dshm\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:13:54.991889 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:54.991856 2573 generic.go:356] "Generic (PLEG): container finished" podID="8fc55f55-7f4f-440e-be40-5255dd100699" containerID="3e7e2945088c65e38e3ebd0f6a5e39f434f5875a8d7edf68225722b0f5080b29" exitCode=0 Aug 05 17:13:54.992076 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:54.991896 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" Aug 05 17:13:54.992076 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:54.991908 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" event={"ID":"8fc55f55-7f4f-440e-be40-5255dd100699","Type":"ContainerDied","Data":"3e7e2945088c65e38e3ebd0f6a5e39f434f5875a8d7edf68225722b0f5080b29"} Aug 05 17:13:54.992076 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:54.991947 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm" event={"ID":"8fc55f55-7f4f-440e-be40-5255dd100699","Type":"ContainerDied","Data":"9d56e1d6f66b774ed04489309dfc309113f18401b9d77ac980725e8c0bd8e1ee"} Aug 05 17:13:54.992076 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:54.991967 2573 scope.go:117] "RemoveContainer" containerID="3e7e2945088c65e38e3ebd0f6a5e39f434f5875a8d7edf68225722b0f5080b29" Aug 05 17:13:55.000721 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:55.000702 2573 scope.go:117] "RemoveContainer" containerID="3e7e2945088c65e38e3ebd0f6a5e39f434f5875a8d7edf68225722b0f5080b29" Aug 05 17:13:55.001007 ip-10-0-132-37 kubenswrapper[2573]: E0805 17:13:55.000984 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e7e2945088c65e38e3ebd0f6a5e39f434f5875a8d7edf68225722b0f5080b29\": container with ID starting with 3e7e2945088c65e38e3ebd0f6a5e39f434f5875a8d7edf68225722b0f5080b29 not found: ID does not exist" containerID="3e7e2945088c65e38e3ebd0f6a5e39f434f5875a8d7edf68225722b0f5080b29" Aug 05 17:13:55.001053 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:55.001015 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e7e2945088c65e38e3ebd0f6a5e39f434f5875a8d7edf68225722b0f5080b29"} err="failed to get container status \"3e7e2945088c65e38e3ebd0f6a5e39f434f5875a8d7edf68225722b0f5080b29\": rpc error: code = NotFound desc = could not find container \"3e7e2945088c65e38e3ebd0f6a5e39f434f5875a8d7edf68225722b0f5080b29\": container with ID starting with 3e7e2945088c65e38e3ebd0f6a5e39f434f5875a8d7edf68225722b0f5080b29 not found: ID does not exist" Aug 05 17:13:55.016584 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:55.016557 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm"] Aug 05 17:13:55.023609 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:55.023586 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-46e6bf/auth-test-model-46e6bf-kserve-6fb96dcb59-j64gm"] Aug 05 17:13:55.310054 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:55.310017 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fc55f55-7f4f-440e-be40-5255dd100699" path="/var/lib/kubelet/pods/8fc55f55-7f4f-440e-be40-5255dd100699/volumes" Aug 05 17:13:59.525328 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:13:59.525289 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:14:13.741768 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:13.741723 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:14:18.307822 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.307785 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr"] Aug 05 17:14:18.308195 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.308095 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fc55f55-7f4f-440e-be40-5255dd100699" containerName="main" Aug 05 17:14:18.308195 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.308106 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fc55f55-7f4f-440e-be40-5255dd100699" containerName="main" Aug 05 17:14:18.308195 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.308166 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fc55f55-7f4f-440e-be40-5255dd100699" containerName="main" Aug 05 17:14:18.311075 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.311057 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" Aug 05 17:14:18.316875 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.316847 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-46e6bf\"/\"default-dockercfg-c2gzl\"" Aug 05 17:14:18.316875 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.316868 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-46e6bf\"/\"sub-test-model-46e6bf-kserve-self-signed-certs\"" Aug 05 17:14:18.317064 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.316875 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-46e6bf\"/\"openshift-service-ca.crt\"" Aug 05 17:14:18.317064 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.316855 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-46e6bf\"/\"kube-root-ca.crt\"" Aug 05 17:14:18.322326 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.322304 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr"] Aug 05 17:14:18.396109 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.396069 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:14:18.440082 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.440049 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5983a9c-43fa-4330-a202-af6c46109089-dshm\") pod \"sub-test-model-46e6bf-kserve-66788654f7-988pr\" (UID: \"a5983a9c-43fa-4330-a202-af6c46109089\") " pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" Aug 05 17:14:18.440256 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.440099 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5983a9c-43fa-4330-a202-af6c46109089-model-cache\") pod \"sub-test-model-46e6bf-kserve-66788654f7-988pr\" (UID: \"a5983a9c-43fa-4330-a202-af6c46109089\") " pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" Aug 05 17:14:18.440256 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.440119 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5983a9c-43fa-4330-a202-af6c46109089-tls-certs\") pod \"sub-test-model-46e6bf-kserve-66788654f7-988pr\" (UID: \"a5983a9c-43fa-4330-a202-af6c46109089\") " pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" Aug 05 17:14:18.440256 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.440167 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5983a9c-43fa-4330-a202-af6c46109089-tmp-dir\") pod \"sub-test-model-46e6bf-kserve-66788654f7-988pr\" (UID: \"a5983a9c-43fa-4330-a202-af6c46109089\") " pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" Aug 05 17:14:18.440256 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.440184 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcz84\" (UniqueName: \"kubernetes.io/projected/a5983a9c-43fa-4330-a202-af6c46109089-kube-api-access-jcz84\") pod \"sub-test-model-46e6bf-kserve-66788654f7-988pr\" (UID: \"a5983a9c-43fa-4330-a202-af6c46109089\") " pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" Aug 05 17:14:18.440256 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.440235 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5983a9c-43fa-4330-a202-af6c46109089-home\") pod \"sub-test-model-46e6bf-kserve-66788654f7-988pr\" (UID: \"a5983a9c-43fa-4330-a202-af6c46109089\") " pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" Aug 05 17:14:18.541427 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.541363 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5983a9c-43fa-4330-a202-af6c46109089-home\") pod \"sub-test-model-46e6bf-kserve-66788654f7-988pr\" (UID: \"a5983a9c-43fa-4330-a202-af6c46109089\") " pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" Aug 05 17:14:18.541604 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.541485 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5983a9c-43fa-4330-a202-af6c46109089-dshm\") pod \"sub-test-model-46e6bf-kserve-66788654f7-988pr\" (UID: \"a5983a9c-43fa-4330-a202-af6c46109089\") " pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" Aug 05 17:14:18.541604 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.541537 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5983a9c-43fa-4330-a202-af6c46109089-model-cache\") pod \"sub-test-model-46e6bf-kserve-66788654f7-988pr\" (UID: \"a5983a9c-43fa-4330-a202-af6c46109089\") " pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" Aug 05 17:14:18.541678 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.541647 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5983a9c-43fa-4330-a202-af6c46109089-tls-certs\") pod \"sub-test-model-46e6bf-kserve-66788654f7-988pr\" (UID: \"a5983a9c-43fa-4330-a202-af6c46109089\") " pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" Aug 05 17:14:18.541717 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.541689 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5983a9c-43fa-4330-a202-af6c46109089-tmp-dir\") pod \"sub-test-model-46e6bf-kserve-66788654f7-988pr\" (UID: \"a5983a9c-43fa-4330-a202-af6c46109089\") " pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" Aug 05 17:14:18.541763 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.541717 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcz84\" (UniqueName: \"kubernetes.io/projected/a5983a9c-43fa-4330-a202-af6c46109089-kube-api-access-jcz84\") pod \"sub-test-model-46e6bf-kserve-66788654f7-988pr\" (UID: \"a5983a9c-43fa-4330-a202-af6c46109089\") " pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" Aug 05 17:14:18.541852 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.541829 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5983a9c-43fa-4330-a202-af6c46109089-home\") pod \"sub-test-model-46e6bf-kserve-66788654f7-988pr\" (UID: \"a5983a9c-43fa-4330-a202-af6c46109089\") " pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" Aug 05 17:14:18.541974 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.541950 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5983a9c-43fa-4330-a202-af6c46109089-model-cache\") pod \"sub-test-model-46e6bf-kserve-66788654f7-988pr\" (UID: \"a5983a9c-43fa-4330-a202-af6c46109089\") " pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" Aug 05 17:14:18.542065 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.542036 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5983a9c-43fa-4330-a202-af6c46109089-tmp-dir\") pod \"sub-test-model-46e6bf-kserve-66788654f7-988pr\" (UID: \"a5983a9c-43fa-4330-a202-af6c46109089\") " pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" Aug 05 17:14:18.543793 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.543774 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5983a9c-43fa-4330-a202-af6c46109089-dshm\") pod \"sub-test-model-46e6bf-kserve-66788654f7-988pr\" (UID: \"a5983a9c-43fa-4330-a202-af6c46109089\") " pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" Aug 05 17:14:18.544053 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.544034 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5983a9c-43fa-4330-a202-af6c46109089-tls-certs\") pod \"sub-test-model-46e6bf-kserve-66788654f7-988pr\" (UID: \"a5983a9c-43fa-4330-a202-af6c46109089\") " pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" Aug 05 17:14:18.555589 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.555567 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcz84\" (UniqueName: \"kubernetes.io/projected/a5983a9c-43fa-4330-a202-af6c46109089-kube-api-access-jcz84\") pod \"sub-test-model-46e6bf-kserve-66788654f7-988pr\" (UID: \"a5983a9c-43fa-4330-a202-af6c46109089\") " pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" Aug 05 17:14:18.620105 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.620019 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" Aug 05 17:14:18.755666 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:18.755633 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr"] Aug 05 17:14:18.758927 ip-10-0-132-37 kubenswrapper[2573]: W0805 17:14:18.758898 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5983a9c_43fa_4330_a202_af6c46109089.slice/crio-8fd2ff5cc5878e8ff4a710cd3f02aa564e5658d353ef620d1d1567a2a27d7d9f WatchSource:0}: Error finding container 8fd2ff5cc5878e8ff4a710cd3f02aa564e5658d353ef620d1d1567a2a27d7d9f: Status 404 returned error can't find the container with id 8fd2ff5cc5878e8ff4a710cd3f02aa564e5658d353ef620d1d1567a2a27d7d9f Aug 05 17:14:19.101137 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:19.101103 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" event={"ID":"a5983a9c-43fa-4330-a202-af6c46109089","Type":"ContainerStarted","Data":"b399170caba0c5de663a720bad57a69ead4d599cdcdfd260cf356413b6733d99"} Aug 05 17:14:19.101137 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:19.101143 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" event={"ID":"a5983a9c-43fa-4330-a202-af6c46109089","Type":"ContainerStarted","Data":"8fd2ff5cc5878e8ff4a710cd3f02aa564e5658d353ef620d1d1567a2a27d7d9f"} Aug 05 17:14:19.125023 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:19.124973 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" podStartSLOduration=1.124957341 podStartE2EDuration="1.124957341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:14:19.123616841 +0000 UTC m=+3174.280462732" watchObservedRunningTime="2026-08-05 17:14:19.124957341 +0000 UTC m=+3174.281803244" Aug 05 17:14:28.620693 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:28.620661 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" Aug 05 17:14:28.621114 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:28.620899 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" Aug 05 17:14:28.633291 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:28.633274 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" Aug 05 17:14:29.157630 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:29.157600 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" Aug 05 17:14:35.189422 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:14:35.189361 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:15:24.262914 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:24.262808 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:15:25.165429 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:25.165398 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr"] Aug 05 17:15:25.187906 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:25.187862 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:15:30.939668 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:30.939633 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:15:31.004694 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:31.004663 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:15:40.183760 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.183714 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" podUID="a5983a9c-43fa-4330-a202-af6c46109089" containerName="main" containerID="cri-o://b399170caba0c5de663a720bad57a69ead4d599cdcdfd260cf356413b6733d99" gracePeriod=45 Aug 05 17:15:40.432941 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.432911 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" Aug 05 17:15:40.470243 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.470163 2573 generic.go:356] "Generic (PLEG): container finished" podID="a5983a9c-43fa-4330-a202-af6c46109089" containerID="b399170caba0c5de663a720bad57a69ead4d599cdcdfd260cf356413b6733d99" exitCode=0 Aug 05 17:15:40.470243 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.470201 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" event={"ID":"a5983a9c-43fa-4330-a202-af6c46109089","Type":"ContainerDied","Data":"b399170caba0c5de663a720bad57a69ead4d599cdcdfd260cf356413b6733d99"} Aug 05 17:15:40.470243 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.470211 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" Aug 05 17:15:40.470243 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.470228 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr" event={"ID":"a5983a9c-43fa-4330-a202-af6c46109089","Type":"ContainerDied","Data":"8fd2ff5cc5878e8ff4a710cd3f02aa564e5658d353ef620d1d1567a2a27d7d9f"} Aug 05 17:15:40.470243 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.470244 2573 scope.go:117] "RemoveContainer" containerID="b399170caba0c5de663a720bad57a69ead4d599cdcdfd260cf356413b6733d99" Aug 05 17:15:40.478485 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.478458 2573 scope.go:117] "RemoveContainer" containerID="b399170caba0c5de663a720bad57a69ead4d599cdcdfd260cf356413b6733d99" Aug 05 17:15:40.478735 ip-10-0-132-37 kubenswrapper[2573]: E0805 17:15:40.478716 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b399170caba0c5de663a720bad57a69ead4d599cdcdfd260cf356413b6733d99\": container with ID starting with b399170caba0c5de663a720bad57a69ead4d599cdcdfd260cf356413b6733d99 not found: ID does not exist" containerID="b399170caba0c5de663a720bad57a69ead4d599cdcdfd260cf356413b6733d99" Aug 05 17:15:40.478801 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.478747 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b399170caba0c5de663a720bad57a69ead4d599cdcdfd260cf356413b6733d99"} err="failed to get container status \"b399170caba0c5de663a720bad57a69ead4d599cdcdfd260cf356413b6733d99\": rpc error: code = NotFound desc = could not find container \"b399170caba0c5de663a720bad57a69ead4d599cdcdfd260cf356413b6733d99\": container with ID starting with b399170caba0c5de663a720bad57a69ead4d599cdcdfd260cf356413b6733d99 not found: ID does not exist" Aug 05 17:15:40.558977 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.558948 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5983a9c-43fa-4330-a202-af6c46109089-tmp-dir\") pod \"a5983a9c-43fa-4330-a202-af6c46109089\" (UID: \"a5983a9c-43fa-4330-a202-af6c46109089\") " Aug 05 17:15:40.558977 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.558983 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5983a9c-43fa-4330-a202-af6c46109089-model-cache\") pod \"a5983a9c-43fa-4330-a202-af6c46109089\" (UID: \"a5983a9c-43fa-4330-a202-af6c46109089\") " Aug 05 17:15:40.559243 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.559002 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5983a9c-43fa-4330-a202-af6c46109089-tls-certs\") pod \"a5983a9c-43fa-4330-a202-af6c46109089\" (UID: \"a5983a9c-43fa-4330-a202-af6c46109089\") " Aug 05 17:15:40.559243 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.559040 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5983a9c-43fa-4330-a202-af6c46109089-dshm\") pod \"a5983a9c-43fa-4330-a202-af6c46109089\" (UID: \"a5983a9c-43fa-4330-a202-af6c46109089\") " Aug 05 17:15:40.559243 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.559057 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5983a9c-43fa-4330-a202-af6c46109089-home\") pod \"a5983a9c-43fa-4330-a202-af6c46109089\" (UID: \"a5983a9c-43fa-4330-a202-af6c46109089\") " Aug 05 17:15:40.559243 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.559156 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcz84\" (UniqueName: \"kubernetes.io/projected/a5983a9c-43fa-4330-a202-af6c46109089-kube-api-access-jcz84\") pod \"a5983a9c-43fa-4330-a202-af6c46109089\" (UID: \"a5983a9c-43fa-4330-a202-af6c46109089\") " Aug 05 17:15:40.559459 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.559259 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5983a9c-43fa-4330-a202-af6c46109089-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5983a9c-43fa-4330-a202-af6c46109089" (UID: "a5983a9c-43fa-4330-a202-af6c46109089"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:15:40.559459 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.559306 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5983a9c-43fa-4330-a202-af6c46109089-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5983a9c-43fa-4330-a202-af6c46109089" (UID: "a5983a9c-43fa-4330-a202-af6c46109089"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:15:40.559599 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.559579 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5983a9c-43fa-4330-a202-af6c46109089-tmp-dir\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:15:40.559654 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.559604 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5983a9c-43fa-4330-a202-af6c46109089-model-cache\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:15:40.559699 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.559667 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5983a9c-43fa-4330-a202-af6c46109089-home" (OuterVolumeSpecName: "home") pod "a5983a9c-43fa-4330-a202-af6c46109089" (UID: "a5983a9c-43fa-4330-a202-af6c46109089"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:15:40.561201 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.561165 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5983a9c-43fa-4330-a202-af6c46109089-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5983a9c-43fa-4330-a202-af6c46109089" (UID: "a5983a9c-43fa-4330-a202-af6c46109089"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:15:40.561320 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.561225 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5983a9c-43fa-4330-a202-af6c46109089-kube-api-access-jcz84" (OuterVolumeSpecName: "kube-api-access-jcz84") pod "a5983a9c-43fa-4330-a202-af6c46109089" (UID: "a5983a9c-43fa-4330-a202-af6c46109089"). InnerVolumeSpecName "kube-api-access-jcz84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:15:40.561320 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.561268 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5983a9c-43fa-4330-a202-af6c46109089-dshm" (OuterVolumeSpecName: "dshm") pod "a5983a9c-43fa-4330-a202-af6c46109089" (UID: "a5983a9c-43fa-4330-a202-af6c46109089"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:15:40.650995 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.650961 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:15:40.660113 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.660083 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5983a9c-43fa-4330-a202-af6c46109089-tls-certs\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:15:40.660113 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.660110 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5983a9c-43fa-4330-a202-af6c46109089-dshm\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:15:40.660260 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.660120 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5983a9c-43fa-4330-a202-af6c46109089-home\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:15:40.660260 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.660128 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcz84\" (UniqueName: \"kubernetes.io/projected/a5983a9c-43fa-4330-a202-af6c46109089-kube-api-access-jcz84\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:15:40.794127 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.794096 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr"] Aug 05 17:15:40.799520 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:40.799495 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-46e6bf/sub-test-model-46e6bf-kserve-66788654f7-988pr"] Aug 05 17:15:41.303699 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:15:41.303670 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5983a9c-43fa-4330-a202-af6c46109089" path="/var/lib/kubelet/pods/a5983a9c-43fa-4330-a202-af6c46109089/volumes" Aug 05 17:16:00.894696 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:00.894661 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h"] Aug 05 17:16:00.895254 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:00.894968 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5983a9c-43fa-4330-a202-af6c46109089" containerName="main" Aug 05 17:16:00.895254 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:00.894979 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5983a9c-43fa-4330-a202-af6c46109089" containerName="main" Aug 05 17:16:00.895254 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:00.895046 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5983a9c-43fa-4330-a202-af6c46109089" containerName="main" Aug 05 17:16:00.899295 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:00.899275 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" Aug 05 17:16:00.903827 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:00.903802 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-46e6bf\"/\"kube-root-ca.crt\"" Aug 05 17:16:00.903950 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:00.903861 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-46e6bf\"/\"rate-test-model-46e6bf-kserve-self-signed-certs\"" Aug 05 17:16:00.903950 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:00.903890 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-46e6bf\"/\"openshift-service-ca.crt\"" Aug 05 17:16:00.904094 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:00.903956 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-46e6bf\"/\"default-dockercfg-c2gzl\"" Aug 05 17:16:00.908272 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:00.908229 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h"] Aug 05 17:16:00.935141 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:00.935105 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:16:01.051394 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:01.051339 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e44834b6-4b9a-4321-88f3-ba212a417579-model-cache\") pod \"rate-test-model-46e6bf-kserve-7686884d4c-dlx5h\" (UID: \"e44834b6-4b9a-4321-88f3-ba212a417579\") " pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" Aug 05 17:16:01.051641 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:01.051574 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e44834b6-4b9a-4321-88f3-ba212a417579-home\") pod \"rate-test-model-46e6bf-kserve-7686884d4c-dlx5h\" (UID: \"e44834b6-4b9a-4321-88f3-ba212a417579\") " pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" Aug 05 17:16:01.051718 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:01.051643 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e44834b6-4b9a-4321-88f3-ba212a417579-tls-certs\") pod \"rate-test-model-46e6bf-kserve-7686884d4c-dlx5h\" (UID: \"e44834b6-4b9a-4321-88f3-ba212a417579\") " pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" Aug 05 17:16:01.051779 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:01.051713 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkc6j\" (UniqueName: \"kubernetes.io/projected/e44834b6-4b9a-4321-88f3-ba212a417579-kube-api-access-lkc6j\") pod \"rate-test-model-46e6bf-kserve-7686884d4c-dlx5h\" (UID: \"e44834b6-4b9a-4321-88f3-ba212a417579\") " pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" Aug 05 17:16:01.051844 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:01.051768 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e44834b6-4b9a-4321-88f3-ba212a417579-dshm\") pod \"rate-test-model-46e6bf-kserve-7686884d4c-dlx5h\" (UID: \"e44834b6-4b9a-4321-88f3-ba212a417579\") " pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" Aug 05 17:16:01.051904 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:01.051850 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e44834b6-4b9a-4321-88f3-ba212a417579-tmp-dir\") pod \"rate-test-model-46e6bf-kserve-7686884d4c-dlx5h\" (UID: \"e44834b6-4b9a-4321-88f3-ba212a417579\") " pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" Aug 05 17:16:01.152697 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:01.152596 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e44834b6-4b9a-4321-88f3-ba212a417579-dshm\") pod \"rate-test-model-46e6bf-kserve-7686884d4c-dlx5h\" (UID: \"e44834b6-4b9a-4321-88f3-ba212a417579\") " pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" Aug 05 17:16:01.152697 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:01.152658 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e44834b6-4b9a-4321-88f3-ba212a417579-tmp-dir\") pod \"rate-test-model-46e6bf-kserve-7686884d4c-dlx5h\" (UID: \"e44834b6-4b9a-4321-88f3-ba212a417579\") " pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" Aug 05 17:16:01.152897 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:01.152720 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e44834b6-4b9a-4321-88f3-ba212a417579-model-cache\") pod \"rate-test-model-46e6bf-kserve-7686884d4c-dlx5h\" (UID: \"e44834b6-4b9a-4321-88f3-ba212a417579\") " pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" Aug 05 17:16:01.152897 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:01.152750 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e44834b6-4b9a-4321-88f3-ba212a417579-home\") pod \"rate-test-model-46e6bf-kserve-7686884d4c-dlx5h\" (UID: \"e44834b6-4b9a-4321-88f3-ba212a417579\") " pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" Aug 05 17:16:01.152897 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:01.152796 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e44834b6-4b9a-4321-88f3-ba212a417579-tls-certs\") pod \"rate-test-model-46e6bf-kserve-7686884d4c-dlx5h\" (UID: \"e44834b6-4b9a-4321-88f3-ba212a417579\") " pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" Aug 05 17:16:01.152897 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:01.152819 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkc6j\" (UniqueName: \"kubernetes.io/projected/e44834b6-4b9a-4321-88f3-ba212a417579-kube-api-access-lkc6j\") pod \"rate-test-model-46e6bf-kserve-7686884d4c-dlx5h\" (UID: \"e44834b6-4b9a-4321-88f3-ba212a417579\") " pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" Aug 05 17:16:01.153177 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:01.153140 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e44834b6-4b9a-4321-88f3-ba212a417579-tmp-dir\") pod \"rate-test-model-46e6bf-kserve-7686884d4c-dlx5h\" (UID: \"e44834b6-4b9a-4321-88f3-ba212a417579\") " pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" Aug 05 17:16:01.153177 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:01.153165 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e44834b6-4b9a-4321-88f3-ba212a417579-model-cache\") pod \"rate-test-model-46e6bf-kserve-7686884d4c-dlx5h\" (UID: \"e44834b6-4b9a-4321-88f3-ba212a417579\") " pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" Aug 05 17:16:01.153331 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:01.153194 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e44834b6-4b9a-4321-88f3-ba212a417579-home\") pod \"rate-test-model-46e6bf-kserve-7686884d4c-dlx5h\" (UID: \"e44834b6-4b9a-4321-88f3-ba212a417579\") " pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" Aug 05 17:16:01.154874 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:01.154846 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e44834b6-4b9a-4321-88f3-ba212a417579-dshm\") pod \"rate-test-model-46e6bf-kserve-7686884d4c-dlx5h\" (UID: \"e44834b6-4b9a-4321-88f3-ba212a417579\") " pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" Aug 05 17:16:01.155135 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:01.155116 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e44834b6-4b9a-4321-88f3-ba212a417579-tls-certs\") pod \"rate-test-model-46e6bf-kserve-7686884d4c-dlx5h\" (UID: \"e44834b6-4b9a-4321-88f3-ba212a417579\") " pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" Aug 05 17:16:01.163130 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:01.163107 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkc6j\" (UniqueName: \"kubernetes.io/projected/e44834b6-4b9a-4321-88f3-ba212a417579-kube-api-access-lkc6j\") pod \"rate-test-model-46e6bf-kserve-7686884d4c-dlx5h\" (UID: \"e44834b6-4b9a-4321-88f3-ba212a417579\") " pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" Aug 05 17:16:01.208928 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:01.208887 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" Aug 05 17:16:01.345098 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:01.345073 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h"] Aug 05 17:16:01.347750 ip-10-0-132-37 kubenswrapper[2573]: W0805 17:16:01.347714 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode44834b6_4b9a_4321_88f3_ba212a417579.slice/crio-95e0b8945783a8d4e643b3a47e717058333c56ca782f6e3065f5978dce7f52e5 WatchSource:0}: Error finding container 95e0b8945783a8d4e643b3a47e717058333c56ca782f6e3065f5978dce7f52e5: Status 404 returned error can't find the container with id 95e0b8945783a8d4e643b3a47e717058333c56ca782f6e3065f5978dce7f52e5 Aug 05 17:16:01.566158 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:01.566124 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" event={"ID":"e44834b6-4b9a-4321-88f3-ba212a417579","Type":"ContainerStarted","Data":"66b842c12c60d9a23462ac9db96adaaef849154edaaf964a0e2f7b59b94c553f"} Aug 05 17:16:01.566158 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:01.566166 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" event={"ID":"e44834b6-4b9a-4321-88f3-ba212a417579","Type":"ContainerStarted","Data":"95e0b8945783a8d4e643b3a47e717058333c56ca782f6e3065f5978dce7f52e5"} Aug 05 17:16:01.588913 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:01.588864 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" podStartSLOduration=1.5888494899999999 podStartE2EDuration="1.58884949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:16:01.587086707 +0000 UTC m=+3276.743932621" watchObservedRunningTime="2026-08-05 17:16:01.58884949 +0000 UTC m=+3276.745695391" Aug 05 17:16:11.209866 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:11.209832 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" Aug 05 17:16:11.210322 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:11.210067 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" Aug 05 17:16:11.222123 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:11.222095 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" Aug 05 17:16:11.620955 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:16:11.620923 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" Aug 05 17:17:18.127906 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:18.127864 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h"] Aug 05 17:17:18.328679 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:18.328637 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:17:18.411250 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:18.411146 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:17:30.678771 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:30.678734 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7497-139681450538880"] Aug 05 17:17:30.684040 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:30.683962 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7497-139681450538880" Aug 05 17:17:30.686613 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:30.686589 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7497-139681450538880"] Aug 05 17:17:30.687664 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:30.687646 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vwmvl\"" Aug 05 17:17:30.830259 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:30.830215 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cfd9\" (UniqueName: \"kubernetes.io/projected/f9ba6473-6a16-4323-99b9-93beccea717f-kube-api-access-2cfd9\") pod \"test-curl-7497-139681450538880\" (UID: \"f9ba6473-6a16-4323-99b9-93beccea717f\") " pod="opendatahub/test-curl-7497-139681450538880" Aug 05 17:17:30.931329 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:30.931239 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cfd9\" (UniqueName: \"kubernetes.io/projected/f9ba6473-6a16-4323-99b9-93beccea717f-kube-api-access-2cfd9\") pod \"test-curl-7497-139681450538880\" (UID: \"f9ba6473-6a16-4323-99b9-93beccea717f\") " pod="opendatahub/test-curl-7497-139681450538880" Aug 05 17:17:30.942473 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:30.942443 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cfd9\" (UniqueName: \"kubernetes.io/projected/f9ba6473-6a16-4323-99b9-93beccea717f-kube-api-access-2cfd9\") pod \"test-curl-7497-139681450538880\" (UID: \"f9ba6473-6a16-4323-99b9-93beccea717f\") " pod="opendatahub/test-curl-7497-139681450538880" Aug 05 17:17:30.995340 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:30.995306 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7497-139681450538880" Aug 05 17:17:31.127959 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:31.127936 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7497-139681450538880"] Aug 05 17:17:31.130683 ip-10-0-132-37 kubenswrapper[2573]: W0805 17:17:31.130655 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9ba6473_6a16_4323_99b9_93beccea717f.slice/crio-96d448011a016065995b77d2a8cec3bf8332eb5d3a965ccf8e4b42ff19cc484e WatchSource:0}: Error finding container 96d448011a016065995b77d2a8cec3bf8332eb5d3a965ccf8e4b42ff19cc484e: Status 404 returned error can't find the container with id 96d448011a016065995b77d2a8cec3bf8332eb5d3a965ccf8e4b42ff19cc484e Aug 05 17:17:31.132476 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:31.132447 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 05 17:17:31.978039 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:31.978004 2573 generic.go:356] "Generic (PLEG): container finished" podID="f9ba6473-6a16-4323-99b9-93beccea717f" containerID="37de3936aae052829227be9f61c48d6371fbbadaf05d3355e1442daa73ec0255" exitCode=0 Aug 05 17:17:31.978430 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:31.978054 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7497-139681450538880" event={"ID":"f9ba6473-6a16-4323-99b9-93beccea717f","Type":"ContainerDied","Data":"37de3936aae052829227be9f61c48d6371fbbadaf05d3355e1442daa73ec0255"} Aug 05 17:17:31.978430 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:31.978103 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7497-139681450538880" event={"ID":"f9ba6473-6a16-4323-99b9-93beccea717f","Type":"ContainerStarted","Data":"96d448011a016065995b77d2a8cec3bf8332eb5d3a965ccf8e4b42ff19cc484e"} Aug 05 17:17:33.116042 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.116017 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7497-139681450538880" Aug 05 17:17:33.139684 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.139659 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7497-139681450538880_f9ba6473-6a16-4323-99b9-93beccea717f/test-curl-7497-139681450538880/0.log" Aug 05 17:17:33.146099 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.146067 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" podUID="e44834b6-4b9a-4321-88f3-ba212a417579" containerName="main" containerID="cri-o://66b842c12c60d9a23462ac9db96adaaef849154edaaf964a0e2f7b59b94c553f" gracePeriod=45 Aug 05 17:17:33.166552 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.166525 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7497-139681450538880"] Aug 05 17:17:33.171146 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.171120 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7497-139681450538880"] Aug 05 17:17:33.253787 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.253695 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cfd9\" (UniqueName: \"kubernetes.io/projected/f9ba6473-6a16-4323-99b9-93beccea717f-kube-api-access-2cfd9\") pod \"f9ba6473-6a16-4323-99b9-93beccea717f\" (UID: \"f9ba6473-6a16-4323-99b9-93beccea717f\") " Aug 05 17:17:33.255878 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.255844 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ba6473-6a16-4323-99b9-93beccea717f-kube-api-access-2cfd9" (OuterVolumeSpecName: "kube-api-access-2cfd9") pod "f9ba6473-6a16-4323-99b9-93beccea717f" (UID: "f9ba6473-6a16-4323-99b9-93beccea717f"). InnerVolumeSpecName "kube-api-access-2cfd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:17:33.304302 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.304271 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9ba6473-6a16-4323-99b9-93beccea717f" path="/var/lib/kubelet/pods/f9ba6473-6a16-4323-99b9-93beccea717f/volumes" Aug 05 17:17:33.354819 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.354791 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cfd9\" (UniqueName: \"kubernetes.io/projected/f9ba6473-6a16-4323-99b9-93beccea717f-kube-api-access-2cfd9\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:17:33.392024 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.392001 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" Aug 05 17:17:33.455982 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.455953 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e44834b6-4b9a-4321-88f3-ba212a417579-tmp-dir\") pod \"e44834b6-4b9a-4321-88f3-ba212a417579\" (UID: \"e44834b6-4b9a-4321-88f3-ba212a417579\") " Aug 05 17:17:33.456137 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.456009 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e44834b6-4b9a-4321-88f3-ba212a417579-model-cache\") pod \"e44834b6-4b9a-4321-88f3-ba212a417579\" (UID: \"e44834b6-4b9a-4321-88f3-ba212a417579\") " Aug 05 17:17:33.456137 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.456056 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e44834b6-4b9a-4321-88f3-ba212a417579-tls-certs\") pod \"e44834b6-4b9a-4321-88f3-ba212a417579\" (UID: \"e44834b6-4b9a-4321-88f3-ba212a417579\") " Aug 05 17:17:33.456137 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.456101 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkc6j\" (UniqueName: \"kubernetes.io/projected/e44834b6-4b9a-4321-88f3-ba212a417579-kube-api-access-lkc6j\") pod \"e44834b6-4b9a-4321-88f3-ba212a417579\" (UID: \"e44834b6-4b9a-4321-88f3-ba212a417579\") " Aug 05 17:17:33.456137 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.456131 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e44834b6-4b9a-4321-88f3-ba212a417579-home\") pod \"e44834b6-4b9a-4321-88f3-ba212a417579\" (UID: \"e44834b6-4b9a-4321-88f3-ba212a417579\") " Aug 05 17:17:33.456312 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.456181 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e44834b6-4b9a-4321-88f3-ba212a417579-dshm\") pod \"e44834b6-4b9a-4321-88f3-ba212a417579\" (UID: \"e44834b6-4b9a-4321-88f3-ba212a417579\") " Aug 05 17:17:33.456312 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.456247 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44834b6-4b9a-4321-88f3-ba212a417579-model-cache" (OuterVolumeSpecName: "model-cache") pod "e44834b6-4b9a-4321-88f3-ba212a417579" (UID: "e44834b6-4b9a-4321-88f3-ba212a417579"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:17:33.456312 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.456255 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44834b6-4b9a-4321-88f3-ba212a417579-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e44834b6-4b9a-4321-88f3-ba212a417579" (UID: "e44834b6-4b9a-4321-88f3-ba212a417579"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:17:33.456459 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.456450 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e44834b6-4b9a-4321-88f3-ba212a417579-tmp-dir\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:17:33.456513 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.456449 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44834b6-4b9a-4321-88f3-ba212a417579-home" (OuterVolumeSpecName: "home") pod "e44834b6-4b9a-4321-88f3-ba212a417579" (UID: "e44834b6-4b9a-4321-88f3-ba212a417579"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:17:33.456513 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.456464 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e44834b6-4b9a-4321-88f3-ba212a417579-model-cache\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:17:33.458270 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.458242 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e44834b6-4b9a-4321-88f3-ba212a417579-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e44834b6-4b9a-4321-88f3-ba212a417579" (UID: "e44834b6-4b9a-4321-88f3-ba212a417579"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:17:33.458270 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.458245 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e44834b6-4b9a-4321-88f3-ba212a417579-kube-api-access-lkc6j" (OuterVolumeSpecName: "kube-api-access-lkc6j") pod "e44834b6-4b9a-4321-88f3-ba212a417579" (UID: "e44834b6-4b9a-4321-88f3-ba212a417579"). InnerVolumeSpecName "kube-api-access-lkc6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:17:33.458270 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.458241 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44834b6-4b9a-4321-88f3-ba212a417579-dshm" (OuterVolumeSpecName: "dshm") pod "e44834b6-4b9a-4321-88f3-ba212a417579" (UID: "e44834b6-4b9a-4321-88f3-ba212a417579"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:17:33.557289 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.557261 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkc6j\" (UniqueName: \"kubernetes.io/projected/e44834b6-4b9a-4321-88f3-ba212a417579-kube-api-access-lkc6j\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:17:33.557289 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.557286 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e44834b6-4b9a-4321-88f3-ba212a417579-home\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:17:33.557289 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.557295 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e44834b6-4b9a-4321-88f3-ba212a417579-dshm\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:17:33.557549 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.557306 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e44834b6-4b9a-4321-88f3-ba212a417579-tls-certs\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:17:33.988969 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.988873 2573 generic.go:356] "Generic (PLEG): container finished" podID="e44834b6-4b9a-4321-88f3-ba212a417579" containerID="66b842c12c60d9a23462ac9db96adaaef849154edaaf964a0e2f7b59b94c553f" exitCode=0 Aug 05 17:17:33.988969 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.988922 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" Aug 05 17:17:33.989177 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.988960 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" event={"ID":"e44834b6-4b9a-4321-88f3-ba212a417579","Type":"ContainerDied","Data":"66b842c12c60d9a23462ac9db96adaaef849154edaaf964a0e2f7b59b94c553f"} Aug 05 17:17:33.989177 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.989010 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h" event={"ID":"e44834b6-4b9a-4321-88f3-ba212a417579","Type":"ContainerDied","Data":"95e0b8945783a8d4e643b3a47e717058333c56ca782f6e3065f5978dce7f52e5"} Aug 05 17:17:33.989177 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.989032 2573 scope.go:117] "RemoveContainer" containerID="66b842c12c60d9a23462ac9db96adaaef849154edaaf964a0e2f7b59b94c553f" Aug 05 17:17:33.990486 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.990211 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7497-139681450538880" Aug 05 17:17:33.998392 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.998353 2573 scope.go:117] "RemoveContainer" containerID="66b842c12c60d9a23462ac9db96adaaef849154edaaf964a0e2f7b59b94c553f" Aug 05 17:17:33.998729 ip-10-0-132-37 kubenswrapper[2573]: E0805 17:17:33.998710 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66b842c12c60d9a23462ac9db96adaaef849154edaaf964a0e2f7b59b94c553f\": container with ID starting with 66b842c12c60d9a23462ac9db96adaaef849154edaaf964a0e2f7b59b94c553f not found: ID does not exist" containerID="66b842c12c60d9a23462ac9db96adaaef849154edaaf964a0e2f7b59b94c553f" Aug 05 17:17:33.998781 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.998738 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b842c12c60d9a23462ac9db96adaaef849154edaaf964a0e2f7b59b94c553f"} err="failed to get container status \"66b842c12c60d9a23462ac9db96adaaef849154edaaf964a0e2f7b59b94c553f\": rpc error: code = NotFound desc = could not find container \"66b842c12c60d9a23462ac9db96adaaef849154edaaf964a0e2f7b59b94c553f\": container with ID starting with 66b842c12c60d9a23462ac9db96adaaef849154edaaf964a0e2f7b59b94c553f not found: ID does not exist" Aug 05 17:17:33.998781 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:33.998754 2573 scope.go:117] "RemoveContainer" containerID="37de3936aae052829227be9f61c48d6371fbbadaf05d3355e1442daa73ec0255" Aug 05 17:17:34.014961 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:34.014938 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h"] Aug 05 17:17:34.018501 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:34.018480 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-46e6bf/rate-test-model-46e6bf-kserve-7686884d4c-dlx5h"] Aug 05 17:17:35.309204 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:35.309172 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e44834b6-4b9a-4321-88f3-ba212a417579" path="/var/lib/kubelet/pods/e44834b6-4b9a-4321-88f3-ba212a417579/volumes" Aug 05 17:17:36.088445 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.088409 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:17:36.384998 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.384910 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7"] Aug 05 17:17:36.385790 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.385771 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e44834b6-4b9a-4321-88f3-ba212a417579" containerName="main" Aug 05 17:17:36.385790 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.385789 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e44834b6-4b9a-4321-88f3-ba212a417579" containerName="main" Aug 05 17:17:36.385921 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.385805 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9ba6473-6a16-4323-99b9-93beccea717f" containerName="test-curl-7497-139681450538880" Aug 05 17:17:36.385921 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.385812 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9ba6473-6a16-4323-99b9-93beccea717f" containerName="test-curl-7497-139681450538880" Aug 05 17:17:36.385921 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.385859 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9ba6473-6a16-4323-99b9-93beccea717f" containerName="test-curl-7497-139681450538880" Aug 05 17:17:36.385921 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.385869 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e44834b6-4b9a-4321-88f3-ba212a417579" containerName="main" Aug 05 17:17:36.390265 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.390235 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.393835 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.393816 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-b2b0ab-openshift-default-dockercfg-rmt75\"" Aug 05 17:17:36.407498 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.407475 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7"] Aug 05 17:17:36.417658 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.417630 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:17:36.485568 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.485526 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-workload-certs\") pod \"e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.485755 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.485587 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-credential-socket\") pod \"e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.485755 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.485624 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-workload-socket\") pod \"e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.485755 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.485653 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istio-data\") pod \"e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.485755 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.485700 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istio-token\") pod \"e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.485755 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.485732 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istio-podinfo\") pod \"e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.486048 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.485780 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fxdp\" (UniqueName: \"kubernetes.io/projected/2405d0d6-c0f2-4c17-bdfc-48ede7762288-kube-api-access-5fxdp\") pod \"e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.486048 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.485832 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istiod-ca-cert\") pod \"e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.486048 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.485862 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istio-envoy\") pod \"e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.586829 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.586778 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istio-podinfo\") pod \"e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.587044 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.586846 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fxdp\" (UniqueName: \"kubernetes.io/projected/2405d0d6-c0f2-4c17-bdfc-48ede7762288-kube-api-access-5fxdp\") pod \"e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.587044 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.586893 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istiod-ca-cert\") pod \"e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.587044 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.586923 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istio-envoy\") pod \"e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.587044 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.586970 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-workload-certs\") pod \"e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.587044 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.587008 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-credential-socket\") pod \"e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.587044 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.587038 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-workload-socket\") pod \"e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.587413 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.587064 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istio-data\") pod \"e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.587413 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.587112 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istio-token\") pod \"e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.587543 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.587482 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istio-data\") pod \"e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.587603 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.587552 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-credential-socket\") pod \"e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.587711 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.587685 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-workload-socket\") pod \"e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.587827 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.587740 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-workload-certs\") pod \"e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.587889 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.587833 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istiod-ca-cert\") pod \"e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.589912 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.589893 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istio-envoy\") pod \"e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.590107 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.590092 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istio-podinfo\") pod \"e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.596104 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.596078 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istio-token\") pod \"e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.596665 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.596642 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fxdp\" (UniqueName: \"kubernetes.io/projected/2405d0d6-c0f2-4c17-bdfc-48ede7762288-kube-api-access-5fxdp\") pod \"e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.700650 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.700547 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:36.849249 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.849209 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7"] Aug 05 17:17:36.849644 ip-10-0-132-37 kubenswrapper[2573]: W0805 17:17:36.849620 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2405d0d6_c0f2_4c17_bdfc_48ede7762288.slice/crio-e9414c21d1184f1bf6ccf02658905f381da6e9a3eda41b202461605767763d0d WatchSource:0}: Error finding container e9414c21d1184f1bf6ccf02658905f381da6e9a3eda41b202461605767763d0d: Status 404 returned error can't find the container with id e9414c21d1184f1bf6ccf02658905f381da6e9a3eda41b202461605767763d0d Aug 05 17:17:36.852010 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.851972 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:17:36.852100 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.852053 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:17:36.852175 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:36.852097 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:17:37.008048 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:37.008012 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" event={"ID":"2405d0d6-c0f2-4c17-bdfc-48ede7762288","Type":"ContainerStarted","Data":"fb6e2aa459405e74e78d41b0bc5192f0a5cf9c3422d755c6ebd8f443d2198313"} Aug 05 17:17:37.008201 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:37.008076 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" event={"ID":"2405d0d6-c0f2-4c17-bdfc-48ede7762288","Type":"ContainerStarted","Data":"e9414c21d1184f1bf6ccf02658905f381da6e9a3eda41b202461605767763d0d"} Aug 05 17:17:37.036863 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:37.036800 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" podStartSLOduration=1.036777881 podStartE2EDuration="1.036777881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:17:37.033866595 +0000 UTC m=+3372.190712498" watchObservedRunningTime="2026-08-05 17:17:37.036777881 +0000 UTC m=+3372.193623783" Aug 05 17:17:37.184866 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:37.184820 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:17:37.701127 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:37.701092 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:37.707317 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:37.707293 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:38.012227 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:38.012192 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:38.013086 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:38.013066 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:39.096962 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:39.096920 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:17:39.251048 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:39.251016 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:17:40.252754 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:40.252713 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:17:40.368148 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:40.368109 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:17:40.849889 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:40.849854 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:17:40.970390 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:40.970335 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:17:53.013911 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:53.013878 2573 patch_prober.go:28] interesting pod/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Aug 05 17:17:53.014324 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:53.013941 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" podUID="2405d0d6-c0f2-4c17-bdfc-48ede7762288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Aug 05 17:17:53.088752 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:53.088721 2573 generic.go:356] "Generic (PLEG): container finished" podID="2405d0d6-c0f2-4c17-bdfc-48ede7762288" containerID="fb6e2aa459405e74e78d41b0bc5192f0a5cf9c3422d755c6ebd8f443d2198313" exitCode=0 Aug 05 17:17:53.088910 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:53.088794 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" event={"ID":"2405d0d6-c0f2-4c17-bdfc-48ede7762288","Type":"ContainerDied","Data":"fb6e2aa459405e74e78d41b0bc5192f0a5cf9c3422d755c6ebd8f443d2198313"} Aug 05 17:17:53.089202 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:53.089186 2573 scope.go:117] "RemoveContainer" containerID="fb6e2aa459405e74e78d41b0bc5192f0a5cf9c3422d755c6ebd8f443d2198313" Aug 05 17:17:53.090187 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:53.090163 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:17:53.090244 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:53.090229 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:17:53.090287 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:53.090264 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:17:54.095955 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:54.095915 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" event={"ID":"2405d0d6-c0f2-4c17-bdfc-48ede7762288","Type":"ContainerStarted","Data":"2f5e0a6a02b62566acad201539ad66eb0a176978e715238cb08c43ff16f76b68"} Aug 05 17:17:54.700778 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:54.700738 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:54.700981 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:54.700952 2573 patch_prober.go:28] interesting pod/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Aug 05 17:17:54.701058 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:54.701025 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" podUID="2405d0d6-c0f2-4c17-bdfc-48ede7762288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Aug 05 17:17:55.701288 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:55.701254 2573 patch_prober.go:28] interesting pod/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Aug 05 17:17:55.701767 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:55.701308 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" podUID="2405d0d6-c0f2-4c17-bdfc-48ede7762288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Aug 05 17:17:56.701613 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:56.701576 2573 patch_prober.go:28] interesting pod/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Aug 05 17:17:56.702016 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:56.701635 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" podUID="2405d0d6-c0f2-4c17-bdfc-48ede7762288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Aug 05 17:17:57.701459 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:57.701418 2573 patch_prober.go:28] interesting pod/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Aug 05 17:17:57.701650 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:57.701480 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" podUID="2405d0d6-c0f2-4c17-bdfc-48ede7762288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Aug 05 17:17:58.701772 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:58.701730 2573 patch_prober.go:28] interesting pod/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Aug 05 17:17:58.702176 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:58.701793 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" podUID="2405d0d6-c0f2-4c17-bdfc-48ede7762288" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Aug 05 17:17:59.705464 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:59.705425 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:59.705952 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:59.705843 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:17:59.706574 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:17:59.706554 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:18:03.997625 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:03.997575 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:18:04.631559 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:04.631510 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:18:04.902614 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:04.902533 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s"] Aug 05 17:18:04.908485 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:04.908466 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:04.911671 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:04.911646 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f6c2cb-openshift-default-dockercfg-5h5hv\"" Aug 05 17:18:04.920193 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:04.920164 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s"] Aug 05 17:18:04.941536 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:04.941504 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:18:04.945428 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:04.945403 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-workload-certs\") pod \"e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:04.945540 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:04.945447 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istiod-ca-cert\") pod \"e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:04.945540 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:04.945517 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhjvl\" (UniqueName: \"kubernetes.io/projected/808215e7-6fbd-4dad-9b36-3e3cf05eec13-kube-api-access-jhjvl\") pod \"e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:04.945627 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:04.945541 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-workload-socket\") pod \"e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:04.945627 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:04.945576 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istio-data\") pod \"e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:04.945627 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:04.945595 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istio-token\") pod \"e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:04.945722 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:04.945693 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-credential-socket\") pod \"e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:04.945763 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:04.945725 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istio-podinfo\") pod \"e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:04.945763 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:04.945747 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istio-envoy\") pod \"e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:05.044652 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:05.044610 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:18:05.046735 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:05.046708 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-credential-socket\") pod \"e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:05.046866 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:05.046736 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istio-podinfo\") pod \"e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:05.046866 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:05.046757 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istio-envoy\") pod \"e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:05.046866 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:05.046791 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-workload-certs\") pod \"e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:05.046866 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:05.046820 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istiod-ca-cert\") pod \"e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:05.046866 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:05.046842 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhjvl\" (UniqueName: \"kubernetes.io/projected/808215e7-6fbd-4dad-9b36-3e3cf05eec13-kube-api-access-jhjvl\") pod \"e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:05.046866 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:05.046865 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-workload-socket\") pod \"e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:05.047204 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:05.046895 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istio-data\") pod \"e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:05.047204 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:05.047174 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istio-token\") pod \"e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:05.047387 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:05.047354 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-workload-socket\") pod \"e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:05.047457 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:05.047407 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istio-data\") pod \"e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:05.047520 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:05.047459 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-credential-socket\") pod \"e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:05.047586 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:05.047566 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-workload-certs\") pod \"e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:05.047713 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:05.047694 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istiod-ca-cert\") pod \"e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:05.049131 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:05.049111 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istio-envoy\") pod \"e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:05.049363 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:05.049346 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istio-podinfo\") pod \"e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:05.057631 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:05.057607 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istio-token\") pod \"e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:05.057720 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:05.057673 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhjvl\" (UniqueName: \"kubernetes.io/projected/808215e7-6fbd-4dad-9b36-3e3cf05eec13-kube-api-access-jhjvl\") pod \"e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:05.219185 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:05.219076 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:05.368088 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:05.368062 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s"] Aug 05 17:18:05.370203 ip-10-0-132-37 kubenswrapper[2573]: W0805 17:18:05.370175 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod808215e7_6fbd_4dad_9b36_3e3cf05eec13.slice/crio-9196859dba3c18ae7cbfee7d11eb766cd793a485e83a649291fa3df7a7db75a7 WatchSource:0}: Error finding container 9196859dba3c18ae7cbfee7d11eb766cd793a485e83a649291fa3df7a7db75a7: Status 404 returned error can't find the container with id 9196859dba3c18ae7cbfee7d11eb766cd793a485e83a649291fa3df7a7db75a7 Aug 05 17:18:05.372477 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:05.372448 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:18:05.372562 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:05.372509 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:18:05.372562 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:05.372539 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:18:06.153114 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:06.153076 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" event={"ID":"808215e7-6fbd-4dad-9b36-3e3cf05eec13","Type":"ContainerStarted","Data":"79d62bac92b5cebae86205364f6283597699b9dd973ea4fbe43ce5887df35a7b"} Aug 05 17:18:06.153516 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:06.153129 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" event={"ID":"808215e7-6fbd-4dad-9b36-3e3cf05eec13","Type":"ContainerStarted","Data":"9196859dba3c18ae7cbfee7d11eb766cd793a485e83a649291fa3df7a7db75a7"} Aug 05 17:18:06.176306 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:06.176254 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" podStartSLOduration=2.176233759 podStartE2EDuration="2.176233759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:18:06.175974937 +0000 UTC m=+3401.332820839" watchObservedRunningTime="2026-08-05 17:18:06.176233759 +0000 UTC m=+3401.333079660" Aug 05 17:18:06.219244 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:06.219207 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:06.224005 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:06.223979 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:07.158406 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:07.158360 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:07.159316 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:07.159298 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:08.086274 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:08.086240 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:18:08.190564 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:08.190529 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:18:09.276949 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:09.276913 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:18:09.376276 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:09.376217 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:18:09.787517 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:09.787479 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:18:09.889538 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:09.889498 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:18:22.159848 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:22.159803 2573 patch_prober.go:28] interesting pod/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Aug 05 17:18:22.160299 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:22.159886 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" podUID="808215e7-6fbd-4dad-9b36-3e3cf05eec13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Aug 05 17:18:22.226833 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:22.226800 2573 generic.go:356] "Generic (PLEG): container finished" podID="808215e7-6fbd-4dad-9b36-3e3cf05eec13" containerID="79d62bac92b5cebae86205364f6283597699b9dd973ea4fbe43ce5887df35a7b" exitCode=0 Aug 05 17:18:22.227008 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:22.226875 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" event={"ID":"808215e7-6fbd-4dad-9b36-3e3cf05eec13","Type":"ContainerDied","Data":"79d62bac92b5cebae86205364f6283597699b9dd973ea4fbe43ce5887df35a7b"} Aug 05 17:18:22.227347 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:22.227326 2573 scope.go:117] "RemoveContainer" containerID="79d62bac92b5cebae86205364f6283597699b9dd973ea4fbe43ce5887df35a7b" Aug 05 17:18:22.228474 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:22.228448 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:18:22.228540 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:22.228517 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:18:22.228576 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:22.228550 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:18:23.244352 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:23.244310 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" event={"ID":"808215e7-6fbd-4dad-9b36-3e3cf05eec13","Type":"ContainerStarted","Data":"8e8bb4df717e6a44ff0814d2c7f6488dd46dfc82e5ccb4c3780116047214a84d"} Aug 05 17:18:24.220068 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:24.220019 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:24.220334 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:24.220308 2573 patch_prober.go:28] interesting pod/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Aug 05 17:18:24.220423 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:24.220402 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" podUID="808215e7-6fbd-4dad-9b36-3e3cf05eec13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Aug 05 17:18:25.219698 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:25.219663 2573 patch_prober.go:28] interesting pod/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Aug 05 17:18:25.220095 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:25.219715 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" podUID="808215e7-6fbd-4dad-9b36-3e3cf05eec13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Aug 05 17:18:26.220096 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:26.220054 2573 patch_prober.go:28] interesting pod/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Aug 05 17:18:26.220526 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:26.220115 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" podUID="808215e7-6fbd-4dad-9b36-3e3cf05eec13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Aug 05 17:18:27.220062 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:27.220018 2573 patch_prober.go:28] interesting pod/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Aug 05 17:18:27.220253 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:27.220089 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" podUID="808215e7-6fbd-4dad-9b36-3e3cf05eec13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Aug 05 17:18:28.219753 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:28.219717 2573 patch_prober.go:28] interesting pod/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Aug 05 17:18:28.219932 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:28.219774 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" podUID="808215e7-6fbd-4dad-9b36-3e3cf05eec13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Aug 05 17:18:29.223973 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:29.223946 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:29.224447 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:29.224288 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:29.225058 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:29.225043 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:18:33.014882 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:33.014844 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:18:53.816771 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:18:53.816734 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:30.196714 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.196676 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:30.495849 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.495768 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg"] Aug 05 17:19:30.499155 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.499132 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.504333 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.504309 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-fc56ab-openshift-default-dockercfg-5dksx\"" Aug 05 17:19:30.525056 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.525030 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg"] Aug 05 17:19:30.529288 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.529259 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-workload-certs\") pod \"e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.529413 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.529307 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xq4z\" (UniqueName: \"kubernetes.io/projected/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-kube-api-access-4xq4z\") pod \"e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.529413 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.529338 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istio-data\") pod \"e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.529538 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.529440 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-workload-socket\") pod \"e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.529596 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.529571 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istio-token\") pod \"e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.529653 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.529623 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istiod-ca-cert\") pod \"e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.529692 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.529661 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-credential-socket\") pod \"e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.529735 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.529687 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istio-podinfo\") pod \"e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.529778 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.529762 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istio-envoy\") pod \"e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.584467 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.570862 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:30.630730 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.630690 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-workload-socket\") pod \"e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.630920 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.630755 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istio-token\") pod \"e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.630920 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.630782 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istiod-ca-cert\") pod \"e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.630920 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.630799 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-credential-socket\") pod \"e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.630920 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.630814 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istio-podinfo\") pod \"e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.630920 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.630835 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istio-envoy\") pod \"e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.630920 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.630876 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-workload-certs\") pod \"e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.630920 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.630905 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xq4z\" (UniqueName: \"kubernetes.io/projected/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-kube-api-access-4xq4z\") pod \"e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.631317 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.630937 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istio-data\") pod \"e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.631317 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.631165 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-workload-socket\") pod \"e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.631317 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.631207 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-credential-socket\") pod \"e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.631317 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.631270 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istio-data\") pod \"e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.631563 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.631487 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-workload-certs\") pod \"e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.631623 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.631600 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istiod-ca-cert\") pod \"e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.633307 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.633285 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istio-envoy\") pod \"e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.633462 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.633310 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istio-podinfo\") pod \"e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.650630 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.650605 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istio-token\") pod \"e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.655363 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.655340 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xq4z\" (UniqueName: \"kubernetes.io/projected/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-kube-api-access-4xq4z\") pod \"e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.810026 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.809995 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:30.851537 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.851504 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:30.952220 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.952197 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg"] Aug 05 17:19:30.954245 ip-10-0-132-37 kubenswrapper[2573]: W0805 17:19:30.954215 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ffa5a86_0a93_4fc8_83df_1f0f368f86e2.slice/crio-5bb2ba2450b1e326d89aab9c3b08e0dd1e0bf62f7978bf072024bbea2bd66bb7 WatchSource:0}: Error finding container 5bb2ba2450b1e326d89aab9c3b08e0dd1e0bf62f7978bf072024bbea2bd66bb7: Status 404 returned error can't find the container with id 5bb2ba2450b1e326d89aab9c3b08e0dd1e0bf62f7978bf072024bbea2bd66bb7 Aug 05 17:19:30.956461 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.956430 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:19:30.956546 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.956498 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:19:30.956546 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:30.956526 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:19:31.560414 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:31.560363 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" event={"ID":"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2","Type":"ContainerStarted","Data":"337b0443af95cfc2149c15f091516068fb7a83f99d958c57bc3d4d79239b709f"} Aug 05 17:19:31.560828 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:31.560437 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" event={"ID":"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2","Type":"ContainerStarted","Data":"5bb2ba2450b1e326d89aab9c3b08e0dd1e0bf62f7978bf072024bbea2bd66bb7"} Aug 05 17:19:31.584862 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:31.584815 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" podStartSLOduration=1.584800715 podStartE2EDuration="1.584800715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:19:31.583144461 +0000 UTC m=+3486.739990389" watchObservedRunningTime="2026-08-05 17:19:31.584800715 +0000 UTC m=+3486.741646615" Aug 05 17:19:31.811181 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:31.811094 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:32.815548 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:32.815519 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:33.465589 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:33.465550 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:33.565239 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:33.565207 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:33.568770 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:33.568741 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:33.569740 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:33.569720 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:34.583233 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:34.583198 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:34.905462 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:34.905354 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:35.119793 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:35.119740 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:35.218642 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:35.218549 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:48.201004 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:48.200963 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:49.610569 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:49.610528 2573 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Aug 05 17:19:49.610971 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:49.610590 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" podUID="2ffa5a86-0a93-4fc8-83df-1f0f368f86e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Aug 05 17:19:50.652441 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:50.652402 2573 generic.go:356] "Generic (PLEG): container finished" podID="2ffa5a86-0a93-4fc8-83df-1f0f368f86e2" containerID="337b0443af95cfc2149c15f091516068fb7a83f99d958c57bc3d4d79239b709f" exitCode=0 Aug 05 17:19:50.652853 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:50.652477 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" event={"ID":"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2","Type":"ContainerDied","Data":"337b0443af95cfc2149c15f091516068fb7a83f99d958c57bc3d4d79239b709f"} Aug 05 17:19:50.652895 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:50.652879 2573 scope.go:117] "RemoveContainer" containerID="337b0443af95cfc2149c15f091516068fb7a83f99d958c57bc3d4d79239b709f" Aug 05 17:19:50.654140 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:50.654110 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:19:50.654260 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:50.654193 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:19:50.654260 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:50.654239 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 05 17:19:51.659497 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:51.659462 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" event={"ID":"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2","Type":"ContainerStarted","Data":"e87b3bef7cc64e6bff7f9e7bc3c8a9888aab48619033e44ac37fb62104eb2b75"} Aug 05 17:19:51.810357 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:51.810318 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:51.810694 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:51.810669 2573 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Aug 05 17:19:51.810746 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:51.810722 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" podUID="2ffa5a86-0a93-4fc8-83df-1f0f368f86e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Aug 05 17:19:52.811070 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:52.811035 2573 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Aug 05 17:19:52.811497 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:52.811091 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" podUID="2ffa5a86-0a93-4fc8-83df-1f0f368f86e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Aug 05 17:19:53.810724 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:53.810681 2573 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Aug 05 17:19:53.810908 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:53.810741 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" podUID="2ffa5a86-0a93-4fc8-83df-1f0f368f86e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Aug 05 17:19:54.407470 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:54.407416 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:54.509437 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:54.509401 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:54.576399 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:54.576346 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:54.810912 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:54.810879 2573 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Aug 05 17:19:54.811096 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:54.810960 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" podUID="2ffa5a86-0a93-4fc8-83df-1f0f368f86e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Aug 05 17:19:54.919567 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:54.919529 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:19:55.811011 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:55.810973 2573 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Aug 05 17:19:55.811445 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:55.811032 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" podUID="2ffa5a86-0a93-4fc8-83df-1f0f368f86e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Aug 05 17:19:56.882522 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:56.882488 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:56.883012 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:56.882865 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:19:56.883583 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:19:56.883559 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:20:00.721638 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:00.721598 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:00.857647 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:00.857607 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:00.959157 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:00.959130 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg"] Aug 05 17:20:00.959433 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:00.959405 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" podUID="2ffa5a86-0a93-4fc8-83df-1f0f368f86e2" containerName="istio-proxy" containerID="cri-o://e87b3bef7cc64e6bff7f9e7bc3c8a9888aab48619033e44ac37fb62104eb2b75" gracePeriod=30 Aug 05 17:20:00.985065 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:00.984981 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:01.222299 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:01.222258 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:06.207925 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.207900 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:20:06.378457 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.378419 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istio-podinfo\") pod \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " Aug 05 17:20:06.378658 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.378469 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istio-data\") pod \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " Aug 05 17:20:06.378658 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.378512 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-credential-socket\") pod \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " Aug 05 17:20:06.378658 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.378531 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istio-envoy\") pod \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " Aug 05 17:20:06.378658 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.378553 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-workload-socket\") pod \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " Aug 05 17:20:06.378658 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.378568 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istio-token\") pod \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " Aug 05 17:20:06.378658 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.378600 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-workload-certs\") pod \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " Aug 05 17:20:06.378658 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.378623 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xq4z\" (UniqueName: \"kubernetes.io/projected/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-kube-api-access-4xq4z\") pod \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " Aug 05 17:20:06.379040 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.378754 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istiod-ca-cert\") pod \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\" (UID: \"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2\") " Aug 05 17:20:06.379040 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.378864 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2ffa5a86-0a93-4fc8-83df-1f0f368f86e2" (UID: "2ffa5a86-0a93-4fc8-83df-1f0f368f86e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:20:06.379040 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.378927 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2ffa5a86-0a93-4fc8-83df-1f0f368f86e2" (UID: "2ffa5a86-0a93-4fc8-83df-1f0f368f86e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:20:06.379208 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.379040 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2ffa5a86-0a93-4fc8-83df-1f0f368f86e2" (UID: "2ffa5a86-0a93-4fc8-83df-1f0f368f86e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:20:06.379208 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.379119 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-credential-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:20:06.379208 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.379141 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-workload-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:20:06.379208 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.379195 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istio-data" (OuterVolumeSpecName: "istio-data") pod "2ffa5a86-0a93-4fc8-83df-1f0f368f86e2" (UID: "2ffa5a86-0a93-4fc8-83df-1f0f368f86e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:20:06.379458 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.379256 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2ffa5a86-0a93-4fc8-83df-1f0f368f86e2" (UID: "2ffa5a86-0a93-4fc8-83df-1f0f368f86e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:20:06.380878 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.380852 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2ffa5a86-0a93-4fc8-83df-1f0f368f86e2" (UID: "2ffa5a86-0a93-4fc8-83df-1f0f368f86e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:20:06.380992 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.380908 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-kube-api-access-4xq4z" (OuterVolumeSpecName: "kube-api-access-4xq4z") pod "2ffa5a86-0a93-4fc8-83df-1f0f368f86e2" (UID: "2ffa5a86-0a93-4fc8-83df-1f0f368f86e2"). InnerVolumeSpecName "kube-api-access-4xq4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:20:06.381056 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.381028 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2ffa5a86-0a93-4fc8-83df-1f0f368f86e2" (UID: "2ffa5a86-0a93-4fc8-83df-1f0f368f86e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 17:20:06.381056 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.381041 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istio-token" (OuterVolumeSpecName: "istio-token") pod "2ffa5a86-0a93-4fc8-83df-1f0f368f86e2" (UID: "2ffa5a86-0a93-4fc8-83df-1f0f368f86e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:20:06.479864 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.479827 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istiod-ca-cert\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:20:06.479864 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.479860 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istio-podinfo\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:20:06.479864 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.479869 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istio-data\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:20:06.480104 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.479877 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istio-envoy\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:20:06.480104 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.479885 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-istio-token\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:20:06.480104 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.479892 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-workload-certs\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:20:06.480104 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.479900 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xq4z\" (UniqueName: \"kubernetes.io/projected/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2-kube-api-access-4xq4z\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:20:06.745565 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.745471 2573 generic.go:356] "Generic (PLEG): container finished" podID="2ffa5a86-0a93-4fc8-83df-1f0f368f86e2" containerID="e87b3bef7cc64e6bff7f9e7bc3c8a9888aab48619033e44ac37fb62104eb2b75" exitCode=0 Aug 05 17:20:06.745565 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.745526 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" Aug 05 17:20:06.745753 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.745557 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" event={"ID":"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2","Type":"ContainerDied","Data":"e87b3bef7cc64e6bff7f9e7bc3c8a9888aab48619033e44ac37fb62104eb2b75"} Aug 05 17:20:06.745753 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.745614 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg" event={"ID":"2ffa5a86-0a93-4fc8-83df-1f0f368f86e2","Type":"ContainerDied","Data":"5bb2ba2450b1e326d89aab9c3b08e0dd1e0bf62f7978bf072024bbea2bd66bb7"} Aug 05 17:20:06.745753 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.745636 2573 scope.go:117] "RemoveContainer" containerID="e87b3bef7cc64e6bff7f9e7bc3c8a9888aab48619033e44ac37fb62104eb2b75" Aug 05 17:20:06.755459 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.755440 2573 scope.go:117] "RemoveContainer" containerID="337b0443af95cfc2149c15f091516068fb7a83f99d958c57bc3d4d79239b709f" Aug 05 17:20:06.763830 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.763808 2573 scope.go:117] "RemoveContainer" containerID="e87b3bef7cc64e6bff7f9e7bc3c8a9888aab48619033e44ac37fb62104eb2b75" Aug 05 17:20:06.764107 ip-10-0-132-37 kubenswrapper[2573]: E0805 17:20:06.764081 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e87b3bef7cc64e6bff7f9e7bc3c8a9888aab48619033e44ac37fb62104eb2b75\": container with ID starting with e87b3bef7cc64e6bff7f9e7bc3c8a9888aab48619033e44ac37fb62104eb2b75 not found: ID does not exist" containerID="e87b3bef7cc64e6bff7f9e7bc3c8a9888aab48619033e44ac37fb62104eb2b75" Aug 05 17:20:06.764214 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.764109 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e87b3bef7cc64e6bff7f9e7bc3c8a9888aab48619033e44ac37fb62104eb2b75"} err="failed to get container status \"e87b3bef7cc64e6bff7f9e7bc3c8a9888aab48619033e44ac37fb62104eb2b75\": rpc error: code = NotFound desc = could not find container \"e87b3bef7cc64e6bff7f9e7bc3c8a9888aab48619033e44ac37fb62104eb2b75\": container with ID starting with e87b3bef7cc64e6bff7f9e7bc3c8a9888aab48619033e44ac37fb62104eb2b75 not found: ID does not exist" Aug 05 17:20:06.764214 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.764127 2573 scope.go:117] "RemoveContainer" containerID="337b0443af95cfc2149c15f091516068fb7a83f99d958c57bc3d4d79239b709f" Aug 05 17:20:06.764396 ip-10-0-132-37 kubenswrapper[2573]: E0805 17:20:06.764360 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"337b0443af95cfc2149c15f091516068fb7a83f99d958c57bc3d4d79239b709f\": container with ID starting with 337b0443af95cfc2149c15f091516068fb7a83f99d958c57bc3d4d79239b709f not found: ID does not exist" containerID="337b0443af95cfc2149c15f091516068fb7a83f99d958c57bc3d4d79239b709f" Aug 05 17:20:06.764476 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.764399 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"337b0443af95cfc2149c15f091516068fb7a83f99d958c57bc3d4d79239b709f"} err="failed to get container status \"337b0443af95cfc2149c15f091516068fb7a83f99d958c57bc3d4d79239b709f\": rpc error: code = NotFound desc = could not find container \"337b0443af95cfc2149c15f091516068fb7a83f99d958c57bc3d4d79239b709f\": container with ID starting with 337b0443af95cfc2149c15f091516068fb7a83f99d958c57bc3d4d79239b709f not found: ID does not exist" Aug 05 17:20:06.772491 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.772468 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg"] Aug 05 17:20:06.779835 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:06.779806 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-fc56ab-openshift-default-7b5b99dd94-wrhpg"] Aug 05 17:20:07.304015 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:07.303986 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ffa5a86-0a93-4fc8-83df-1f0f368f86e2" path="/var/lib/kubelet/pods/2ffa5a86-0a93-4fc8-83df-1f0f368f86e2/volumes" Aug 05 17:20:14.943433 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:14.943392 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:15.081687 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:15.081650 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:15.244937 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:15.244841 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:15.348913 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:15.348878 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:15.643268 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:15.643225 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:20.673261 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:20.673227 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:31.636779 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:31.636743 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:57.801219 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:57.801185 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:20:58.334348 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:20:58.334314 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:04.041394 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:04.041336 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:04.138444 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:04.138401 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:04.296093 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:04.296011 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7"] Aug 05 17:21:04.296243 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:04.296213 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" podUID="2405d0d6-c0f2-4c17-bdfc-48ede7762288" containerName="istio-proxy" containerID="cri-o://2f5e0a6a02b62566acad201539ad66eb0a176978e715238cb08c43ff16f76b68" gracePeriod=30 Aug 05 17:21:04.308368 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:04.308338 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:04.650237 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:04.650148 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:09.547165 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:09.547140 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:21:09.666222 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:09.666107 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istio-envoy\") pod \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " Aug 05 17:21:09.666222 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:09.666161 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istio-data\") pod \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " Aug 05 17:21:09.666222 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:09.666185 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istio-podinfo\") pod \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " Aug 05 17:21:09.666565 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:09.666236 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-workload-socket\") pod \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " Aug 05 17:21:09.666565 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:09.666262 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fxdp\" (UniqueName: \"kubernetes.io/projected/2405d0d6-c0f2-4c17-bdfc-48ede7762288-kube-api-access-5fxdp\") pod \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " Aug 05 17:21:09.666565 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:09.666369 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-credential-socket\") pod \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " Aug 05 17:21:09.666565 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:09.666483 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istio-token\") pod \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " Aug 05 17:21:09.666565 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:09.666514 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-workload-certs\") pod \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " Aug 05 17:21:09.666843 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:09.666556 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2405d0d6-c0f2-4c17-bdfc-48ede7762288" (UID: "2405d0d6-c0f2-4c17-bdfc-48ede7762288"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:21:09.666843 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:09.666569 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istio-data" (OuterVolumeSpecName: "istio-data") pod "2405d0d6-c0f2-4c17-bdfc-48ede7762288" (UID: "2405d0d6-c0f2-4c17-bdfc-48ede7762288"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:21:09.666843 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:09.666571 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istiod-ca-cert\") pod \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\" (UID: \"2405d0d6-c0f2-4c17-bdfc-48ede7762288\") " Aug 05 17:21:09.666843 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:09.666784 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2405d0d6-c0f2-4c17-bdfc-48ede7762288" (UID: "2405d0d6-c0f2-4c17-bdfc-48ede7762288"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:21:09.667056 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:09.666865 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2405d0d6-c0f2-4c17-bdfc-48ede7762288" (UID: "2405d0d6-c0f2-4c17-bdfc-48ede7762288"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:21:09.667056 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:09.666928 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2405d0d6-c0f2-4c17-bdfc-48ede7762288" (UID: "2405d0d6-c0f2-4c17-bdfc-48ede7762288"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:21:09.667056 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:09.666983 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istiod-ca-cert\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:21:09.667056 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:09.667001 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istio-data\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:21:09.667056 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:09.667014 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-workload-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:21:09.667056 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:09.667027 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-workload-certs\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:21:09.668397 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:09.668353 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2405d0d6-c0f2-4c17-bdfc-48ede7762288" (UID: "2405d0d6-c0f2-4c17-bdfc-48ede7762288"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:21:09.668562 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:09.668547 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2405d0d6-c0f2-4c17-bdfc-48ede7762288-kube-api-access-5fxdp" (OuterVolumeSpecName: "kube-api-access-5fxdp") pod "2405d0d6-c0f2-4c17-bdfc-48ede7762288" (UID: "2405d0d6-c0f2-4c17-bdfc-48ede7762288"). InnerVolumeSpecName "kube-api-access-5fxdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:21:09.668793 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:09.668775 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2405d0d6-c0f2-4c17-bdfc-48ede7762288" (UID: "2405d0d6-c0f2-4c17-bdfc-48ede7762288"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 17:21:09.668860 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:09.668791 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istio-token" (OuterVolumeSpecName: "istio-token") pod "2405d0d6-c0f2-4c17-bdfc-48ede7762288" (UID: "2405d0d6-c0f2-4c17-bdfc-48ede7762288"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:21:09.767780 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:09.767741 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istio-envoy\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:21:09.767780 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:09.767773 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istio-podinfo\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:21:09.767780 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:09.767783 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fxdp\" (UniqueName: \"kubernetes.io/projected/2405d0d6-c0f2-4c17-bdfc-48ede7762288-kube-api-access-5fxdp\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:21:09.767780 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:09.767794 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2405d0d6-c0f2-4c17-bdfc-48ede7762288-credential-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:21:09.768056 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:09.767802 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2405d0d6-c0f2-4c17-bdfc-48ede7762288-istio-token\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:21:10.049536 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:10.049505 2573 generic.go:356] "Generic (PLEG): container finished" podID="2405d0d6-c0f2-4c17-bdfc-48ede7762288" containerID="2f5e0a6a02b62566acad201539ad66eb0a176978e715238cb08c43ff16f76b68" exitCode=0 Aug 05 17:21:10.049720 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:10.049572 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" Aug 05 17:21:10.049720 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:10.049595 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" event={"ID":"2405d0d6-c0f2-4c17-bdfc-48ede7762288","Type":"ContainerDied","Data":"2f5e0a6a02b62566acad201539ad66eb0a176978e715238cb08c43ff16f76b68"} Aug 05 17:21:10.049720 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:10.049670 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7" event={"ID":"2405d0d6-c0f2-4c17-bdfc-48ede7762288","Type":"ContainerDied","Data":"e9414c21d1184f1bf6ccf02658905f381da6e9a3eda41b202461605767763d0d"} Aug 05 17:21:10.049720 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:10.049693 2573 scope.go:117] "RemoveContainer" containerID="2f5e0a6a02b62566acad201539ad66eb0a176978e715238cb08c43ff16f76b68" Aug 05 17:21:10.059061 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:10.059045 2573 scope.go:117] "RemoveContainer" containerID="fb6e2aa459405e74e78d41b0bc5192f0a5cf9c3422d755c6ebd8f443d2198313" Aug 05 17:21:10.066511 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:10.066494 2573 scope.go:117] "RemoveContainer" containerID="2f5e0a6a02b62566acad201539ad66eb0a176978e715238cb08c43ff16f76b68" Aug 05 17:21:10.066758 ip-10-0-132-37 kubenswrapper[2573]: E0805 17:21:10.066736 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f5e0a6a02b62566acad201539ad66eb0a176978e715238cb08c43ff16f76b68\": container with ID starting with 2f5e0a6a02b62566acad201539ad66eb0a176978e715238cb08c43ff16f76b68 not found: ID does not exist" containerID="2f5e0a6a02b62566acad201539ad66eb0a176978e715238cb08c43ff16f76b68" Aug 05 17:21:10.066852 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:10.066764 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f5e0a6a02b62566acad201539ad66eb0a176978e715238cb08c43ff16f76b68"} err="failed to get container status \"2f5e0a6a02b62566acad201539ad66eb0a176978e715238cb08c43ff16f76b68\": rpc error: code = NotFound desc = could not find container \"2f5e0a6a02b62566acad201539ad66eb0a176978e715238cb08c43ff16f76b68\": container with ID starting with 2f5e0a6a02b62566acad201539ad66eb0a176978e715238cb08c43ff16f76b68 not found: ID does not exist" Aug 05 17:21:10.066852 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:10.066781 2573 scope.go:117] "RemoveContainer" containerID="fb6e2aa459405e74e78d41b0bc5192f0a5cf9c3422d755c6ebd8f443d2198313" Aug 05 17:21:10.066996 ip-10-0-132-37 kubenswrapper[2573]: E0805 17:21:10.066982 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb6e2aa459405e74e78d41b0bc5192f0a5cf9c3422d755c6ebd8f443d2198313\": container with ID starting with fb6e2aa459405e74e78d41b0bc5192f0a5cf9c3422d755c6ebd8f443d2198313 not found: ID does not exist" containerID="fb6e2aa459405e74e78d41b0bc5192f0a5cf9c3422d755c6ebd8f443d2198313" Aug 05 17:21:10.067031 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:10.067000 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6e2aa459405e74e78d41b0bc5192f0a5cf9c3422d755c6ebd8f443d2198313"} err="failed to get container status \"fb6e2aa459405e74e78d41b0bc5192f0a5cf9c3422d755c6ebd8f443d2198313\": rpc error: code = NotFound desc = could not find container \"fb6e2aa459405e74e78d41b0bc5192f0a5cf9c3422d755c6ebd8f443d2198313\": container with ID starting with fb6e2aa459405e74e78d41b0bc5192f0a5cf9c3422d755c6ebd8f443d2198313 not found: ID does not exist" Aug 05 17:21:10.079564 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:10.079541 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7"] Aug 05 17:21:10.084479 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:10.084458 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-b2b0ab-openshift-default-ff97cd6dd-8jzq7"] Aug 05 17:21:11.059678 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:11.059634 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:11.304310 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:11.304279 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2405d0d6-c0f2-4c17-bdfc-48ede7762288" path="/var/lib/kubelet/pods/2405d0d6-c0f2-4c17-bdfc-48ede7762288/volumes" Aug 05 17:21:11.507838 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:11.507810 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:15.359462 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:15.359409 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:15.468943 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:15.468905 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:15.651552 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:15.651457 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:15.886393 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:15.886329 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:16.197769 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:16.197736 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:21.959193 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:21.959100 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:22.026848 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:22.026817 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:22.196180 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:22.196127 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s"] Aug 05 17:21:22.199986 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:22.199946 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" podUID="808215e7-6fbd-4dad-9b36-3e3cf05eec13" containerName="istio-proxy" containerID="cri-o://8e8bb4df717e6a44ff0814d2c7f6488dd46dfc82e5ccb4c3780116047214a84d" gracePeriod=30 Aug 05 17:21:22.218838 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:22.218761 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:22.432670 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:22.432631 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:27.377996 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.377963 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:27.452600 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.452575 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:21:27.633253 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.633218 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istio-data\") pod \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " Aug 05 17:21:27.633506 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.633280 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istiod-ca-cert\") pod \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " Aug 05 17:21:27.633506 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.633303 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istio-podinfo\") pod \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " Aug 05 17:21:27.633506 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.633337 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhjvl\" (UniqueName: \"kubernetes.io/projected/808215e7-6fbd-4dad-9b36-3e3cf05eec13-kube-api-access-jhjvl\") pod \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " Aug 05 17:21:27.633684 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.633508 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-credential-socket\") pod \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " Aug 05 17:21:27.633684 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.633549 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istio-token\") pod \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " Aug 05 17:21:27.633684 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.633572 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istio-envoy\") pod \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " Aug 05 17:21:27.633684 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.633585 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istio-data" (OuterVolumeSpecName: "istio-data") pod "808215e7-6fbd-4dad-9b36-3e3cf05eec13" (UID: "808215e7-6fbd-4dad-9b36-3e3cf05eec13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:21:27.633684 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.633628 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-workload-certs\") pod \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " Aug 05 17:21:27.633684 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.633646 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-workload-socket\") pod \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\" (UID: \"808215e7-6fbd-4dad-9b36-3e3cf05eec13\") " Aug 05 17:21:27.633988 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.633784 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "808215e7-6fbd-4dad-9b36-3e3cf05eec13" (UID: "808215e7-6fbd-4dad-9b36-3e3cf05eec13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:21:27.633988 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.633928 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "808215e7-6fbd-4dad-9b36-3e3cf05eec13" (UID: "808215e7-6fbd-4dad-9b36-3e3cf05eec13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:21:27.634113 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.634059 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "808215e7-6fbd-4dad-9b36-3e3cf05eec13" (UID: "808215e7-6fbd-4dad-9b36-3e3cf05eec13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:21:27.634113 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.634105 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "808215e7-6fbd-4dad-9b36-3e3cf05eec13" (UID: "808215e7-6fbd-4dad-9b36-3e3cf05eec13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:21:27.634225 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.634118 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istiod-ca-cert\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:21:27.634225 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.634137 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-workload-certs\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:21:27.634225 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.634149 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istio-data\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:21:27.635779 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.635749 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/808215e7-6fbd-4dad-9b36-3e3cf05eec13-kube-api-access-jhjvl" (OuterVolumeSpecName: "kube-api-access-jhjvl") pod "808215e7-6fbd-4dad-9b36-3e3cf05eec13" (UID: "808215e7-6fbd-4dad-9b36-3e3cf05eec13"). InnerVolumeSpecName "kube-api-access-jhjvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:21:27.635779 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.635766 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "808215e7-6fbd-4dad-9b36-3e3cf05eec13" (UID: "808215e7-6fbd-4dad-9b36-3e3cf05eec13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:21:27.636022 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.635997 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "808215e7-6fbd-4dad-9b36-3e3cf05eec13" (UID: "808215e7-6fbd-4dad-9b36-3e3cf05eec13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 17:21:27.636117 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.636020 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istio-token" (OuterVolumeSpecName: "istio-token") pod "808215e7-6fbd-4dad-9b36-3e3cf05eec13" (UID: "808215e7-6fbd-4dad-9b36-3e3cf05eec13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:21:27.734719 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.734680 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istio-podinfo\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:21:27.734719 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.734712 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhjvl\" (UniqueName: \"kubernetes.io/projected/808215e7-6fbd-4dad-9b36-3e3cf05eec13-kube-api-access-jhjvl\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:21:27.734719 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.734723 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-credential-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:21:27.734719 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.734732 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istio-token\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:21:27.734985 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.734740 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-istio-envoy\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:21:27.734985 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:27.734749 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/808215e7-6fbd-4dad-9b36-3e3cf05eec13-workload-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:21:28.133119 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:28.133082 2573 generic.go:356] "Generic (PLEG): container finished" podID="808215e7-6fbd-4dad-9b36-3e3cf05eec13" containerID="8e8bb4df717e6a44ff0814d2c7f6488dd46dfc82e5ccb4c3780116047214a84d" exitCode=0 Aug 05 17:21:28.133300 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:28.133117 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" event={"ID":"808215e7-6fbd-4dad-9b36-3e3cf05eec13","Type":"ContainerDied","Data":"8e8bb4df717e6a44ff0814d2c7f6488dd46dfc82e5ccb4c3780116047214a84d"} Aug 05 17:21:28.133300 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:28.133173 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" event={"ID":"808215e7-6fbd-4dad-9b36-3e3cf05eec13","Type":"ContainerDied","Data":"9196859dba3c18ae7cbfee7d11eb766cd793a485e83a649291fa3df7a7db75a7"} Aug 05 17:21:28.133300 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:28.133195 2573 scope.go:117] "RemoveContainer" containerID="8e8bb4df717e6a44ff0814d2c7f6488dd46dfc82e5ccb4c3780116047214a84d" Aug 05 17:21:28.133300 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:28.133133 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s" Aug 05 17:21:28.142786 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:28.142771 2573 scope.go:117] "RemoveContainer" containerID="79d62bac92b5cebae86205364f6283597699b9dd973ea4fbe43ce5887df35a7b" Aug 05 17:21:28.150409 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:28.150391 2573 scope.go:117] "RemoveContainer" containerID="8e8bb4df717e6a44ff0814d2c7f6488dd46dfc82e5ccb4c3780116047214a84d" Aug 05 17:21:28.150631 ip-10-0-132-37 kubenswrapper[2573]: E0805 17:21:28.150616 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e8bb4df717e6a44ff0814d2c7f6488dd46dfc82e5ccb4c3780116047214a84d\": container with ID starting with 8e8bb4df717e6a44ff0814d2c7f6488dd46dfc82e5ccb4c3780116047214a84d not found: ID does not exist" containerID="8e8bb4df717e6a44ff0814d2c7f6488dd46dfc82e5ccb4c3780116047214a84d" Aug 05 17:21:28.150677 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:28.150638 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e8bb4df717e6a44ff0814d2c7f6488dd46dfc82e5ccb4c3780116047214a84d"} err="failed to get container status \"8e8bb4df717e6a44ff0814d2c7f6488dd46dfc82e5ccb4c3780116047214a84d\": rpc error: code = NotFound desc = could not find container \"8e8bb4df717e6a44ff0814d2c7f6488dd46dfc82e5ccb4c3780116047214a84d\": container with ID starting with 8e8bb4df717e6a44ff0814d2c7f6488dd46dfc82e5ccb4c3780116047214a84d not found: ID does not exist" Aug 05 17:21:28.150677 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:28.150653 2573 scope.go:117] "RemoveContainer" containerID="79d62bac92b5cebae86205364f6283597699b9dd973ea4fbe43ce5887df35a7b" Aug 05 17:21:28.150894 ip-10-0-132-37 kubenswrapper[2573]: E0805 17:21:28.150877 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79d62bac92b5cebae86205364f6283597699b9dd973ea4fbe43ce5887df35a7b\": container with ID starting with 79d62bac92b5cebae86205364f6283597699b9dd973ea4fbe43ce5887df35a7b not found: ID does not exist" containerID="79d62bac92b5cebae86205364f6283597699b9dd973ea4fbe43ce5887df35a7b" Aug 05 17:21:28.150951 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:28.150898 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79d62bac92b5cebae86205364f6283597699b9dd973ea4fbe43ce5887df35a7b"} err="failed to get container status \"79d62bac92b5cebae86205364f6283597699b9dd973ea4fbe43ce5887df35a7b\": rpc error: code = NotFound desc = could not find container \"79d62bac92b5cebae86205364f6283597699b9dd973ea4fbe43ce5887df35a7b\": container with ID starting with 79d62bac92b5cebae86205364f6283597699b9dd973ea4fbe43ce5887df35a7b not found: ID does not exist" Aug 05 17:21:28.158710 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:28.158690 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s"] Aug 05 17:21:28.164016 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:28.163995 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f6c2cb-openshift-default-549bdfb8b4-85j7s"] Aug 05 17:21:29.303873 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:29.303843 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="808215e7-6fbd-4dad-9b36-3e3cf05eec13" path="/var/lib/kubelet/pods/808215e7-6fbd-4dad-9b36-3e3cf05eec13/volumes" Aug 05 17:21:33.147535 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:33.147490 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:33.248905 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:33.248804 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:33.347271 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:33.347204 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:33.430561 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:33.430508 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:33.792619 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:33.792588 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:39.905159 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:39.905116 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:40.009271 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:40.009221 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:40.142096 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:40.142065 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw"] Aug 05 17:21:40.142306 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:40.142284 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" podUID="265e37c0-bc0b-421c-bbfa-2e7c74d7d731" containerName="istio-proxy" containerID="cri-o://3d0926befb0a8da1e78f1bf30e73f94ffe5e625cfee0b1e66c97ad64654c49da" gracePeriod=30 Aug 05 17:21:40.177949 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:40.177864 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:40.389937 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:40.389896 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:45.387519 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:45.387493 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:21:45.483708 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:45.483629 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-credential-socket\") pod \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " Aug 05 17:21:45.483708 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:45.483684 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istio-data\") pod \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " Aug 05 17:21:45.483708 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:45.483702 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istio-token\") pod \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " Aug 05 17:21:45.483982 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:45.483725 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istiod-ca-cert\") pod \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " Aug 05 17:21:45.483982 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:45.483765 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-workload-certs\") pod \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " Aug 05 17:21:45.483982 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:45.483815 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkzcg\" (UniqueName: \"kubernetes.io/projected/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-kube-api-access-kkzcg\") pod \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " Aug 05 17:21:45.483982 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:45.483831 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istio-podinfo\") pod \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " Aug 05 17:21:45.483982 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:45.483845 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-workload-socket\") pod \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " Aug 05 17:21:45.483982 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:45.483879 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istio-envoy\") pod \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\" (UID: \"265e37c0-bc0b-421c-bbfa-2e7c74d7d731\") " Aug 05 17:21:45.483982 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:45.483922 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "265e37c0-bc0b-421c-bbfa-2e7c74d7d731" (UID: "265e37c0-bc0b-421c-bbfa-2e7c74d7d731"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:21:45.484325 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:45.484010 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istio-data" (OuterVolumeSpecName: "istio-data") pod "265e37c0-bc0b-421c-bbfa-2e7c74d7d731" (UID: "265e37c0-bc0b-421c-bbfa-2e7c74d7d731"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:21:45.484325 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:45.484153 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-credential-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:21:45.484325 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:45.484204 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istio-data\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:21:45.484325 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:45.484215 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "265e37c0-bc0b-421c-bbfa-2e7c74d7d731" (UID: "265e37c0-bc0b-421c-bbfa-2e7c74d7d731"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:21:45.484325 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:45.484245 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "265e37c0-bc0b-421c-bbfa-2e7c74d7d731" (UID: "265e37c0-bc0b-421c-bbfa-2e7c74d7d731"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:21:45.484606 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:45.484455 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "265e37c0-bc0b-421c-bbfa-2e7c74d7d731" (UID: "265e37c0-bc0b-421c-bbfa-2e7c74d7d731"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:21:45.486025 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:45.485999 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-kube-api-access-kkzcg" (OuterVolumeSpecName: "kube-api-access-kkzcg") pod "265e37c0-bc0b-421c-bbfa-2e7c74d7d731" (UID: "265e37c0-bc0b-421c-bbfa-2e7c74d7d731"). InnerVolumeSpecName "kube-api-access-kkzcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:21:45.486140 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:45.486046 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istio-token" (OuterVolumeSpecName: "istio-token") pod "265e37c0-bc0b-421c-bbfa-2e7c74d7d731" (UID: "265e37c0-bc0b-421c-bbfa-2e7c74d7d731"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:21:45.486329 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:45.486310 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "265e37c0-bc0b-421c-bbfa-2e7c74d7d731" (UID: "265e37c0-bc0b-421c-bbfa-2e7c74d7d731"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:21:45.486329 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:45.486315 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "265e37c0-bc0b-421c-bbfa-2e7c74d7d731" (UID: "265e37c0-bc0b-421c-bbfa-2e7c74d7d731"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 17:21:45.585049 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:45.585015 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istio-token\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:21:45.585049 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:45.585041 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istiod-ca-cert\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:21:45.585049 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:45.585049 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-workload-certs\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:21:45.585275 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:45.585057 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkzcg\" (UniqueName: \"kubernetes.io/projected/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-kube-api-access-kkzcg\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:21:45.585275 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:45.585067 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istio-podinfo\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:21:45.585275 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:45.585075 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-workload-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:21:45.585275 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:45.585084 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/265e37c0-bc0b-421c-bbfa-2e7c74d7d731-istio-envoy\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:21:46.217949 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:46.217915 2573 generic.go:356] "Generic (PLEG): container finished" podID="265e37c0-bc0b-421c-bbfa-2e7c74d7d731" containerID="3d0926befb0a8da1e78f1bf30e73f94ffe5e625cfee0b1e66c97ad64654c49da" exitCode=0 Aug 05 17:21:46.218135 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:46.217964 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" Aug 05 17:21:46.218135 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:46.217985 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" event={"ID":"265e37c0-bc0b-421c-bbfa-2e7c74d7d731","Type":"ContainerDied","Data":"3d0926befb0a8da1e78f1bf30e73f94ffe5e625cfee0b1e66c97ad64654c49da"} Aug 05 17:21:46.218135 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:46.218024 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw" event={"ID":"265e37c0-bc0b-421c-bbfa-2e7c74d7d731","Type":"ContainerDied","Data":"f77385288ab73e204bcc87446bb23af24d489d499eedefeba7f3249d2fbcb932"} Aug 05 17:21:46.218135 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:46.218045 2573 scope.go:117] "RemoveContainer" containerID="3d0926befb0a8da1e78f1bf30e73f94ffe5e625cfee0b1e66c97ad64654c49da" Aug 05 17:21:46.226822 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:46.226804 2573 scope.go:117] "RemoveContainer" containerID="85caf50cac15164e880f6bf4e4e3999d66d07a79891f31e205fde121c9023834" Aug 05 17:21:46.234092 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:46.234074 2573 scope.go:117] "RemoveContainer" containerID="3d0926befb0a8da1e78f1bf30e73f94ffe5e625cfee0b1e66c97ad64654c49da" Aug 05 17:21:46.234324 ip-10-0-132-37 kubenswrapper[2573]: E0805 17:21:46.234305 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d0926befb0a8da1e78f1bf30e73f94ffe5e625cfee0b1e66c97ad64654c49da\": container with ID starting with 3d0926befb0a8da1e78f1bf30e73f94ffe5e625cfee0b1e66c97ad64654c49da not found: ID does not exist" containerID="3d0926befb0a8da1e78f1bf30e73f94ffe5e625cfee0b1e66c97ad64654c49da" Aug 05 17:21:46.234390 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:46.234332 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d0926befb0a8da1e78f1bf30e73f94ffe5e625cfee0b1e66c97ad64654c49da"} err="failed to get container status \"3d0926befb0a8da1e78f1bf30e73f94ffe5e625cfee0b1e66c97ad64654c49da\": rpc error: code = NotFound desc = could not find container \"3d0926befb0a8da1e78f1bf30e73f94ffe5e625cfee0b1e66c97ad64654c49da\": container with ID starting with 3d0926befb0a8da1e78f1bf30e73f94ffe5e625cfee0b1e66c97ad64654c49da not found: ID does not exist" Aug 05 17:21:46.234390 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:46.234349 2573 scope.go:117] "RemoveContainer" containerID="85caf50cac15164e880f6bf4e4e3999d66d07a79891f31e205fde121c9023834" Aug 05 17:21:46.234606 ip-10-0-132-37 kubenswrapper[2573]: E0805 17:21:46.234593 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85caf50cac15164e880f6bf4e4e3999d66d07a79891f31e205fde121c9023834\": container with ID starting with 85caf50cac15164e880f6bf4e4e3999d66d07a79891f31e205fde121c9023834 not found: ID does not exist" containerID="85caf50cac15164e880f6bf4e4e3999d66d07a79891f31e205fde121c9023834" Aug 05 17:21:46.234645 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:46.234608 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85caf50cac15164e880f6bf4e4e3999d66d07a79891f31e205fde121c9023834"} err="failed to get container status \"85caf50cac15164e880f6bf4e4e3999d66d07a79891f31e205fde121c9023834\": rpc error: code = NotFound desc = could not find container \"85caf50cac15164e880f6bf4e4e3999d66d07a79891f31e205fde121c9023834\": container with ID starting with 85caf50cac15164e880f6bf4e4e3999d66d07a79891f31e205fde121c9023834 not found: ID does not exist" Aug 05 17:21:46.240503 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:46.240481 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw"] Aug 05 17:21:46.245151 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:46.245127 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-02b433-openshift-default-86bf8dc7fb-9m9kw"] Aug 05 17:21:47.304054 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:47.304020 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="265e37c0-bc0b-421c-bbfa-2e7c74d7d731" path="/var/lib/kubelet/pods/265e37c0-bc0b-421c-bbfa-2e7c74d7d731/volumes" Aug 05 17:21:50.976988 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:50.976952 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:51.058543 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:51.058512 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:51.166782 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:51.166748 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:51.260110 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:51.260075 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:51.564757 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:51.564661 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:57.691533 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:57.691491 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:57.757333 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:57.757294 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:57.911355 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:57.911324 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp"] Aug 05 17:21:57.911620 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:57.911592 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" podUID="ec6581e2-a031-44ce-860f-de3dd9c45642" containerName="istio-proxy" containerID="cri-o://865cf0e0ff280ebc3aa84057834b58ceef0cedc88fe538e8b7f535c4389d8004" gracePeriod=30 Aug 05 17:21:57.939648 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:57.939611 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:58.160321 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:58.160285 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:21:58.593060 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:21:58.593026 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6cd787c69b-588vp_7f30f2b9-98de-4d12-846e-c3511d31201c/authorino/0.log" Aug 05 17:22:01.456813 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:01.456774 2573 patch_prober.go:28] interesting pod/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 05 17:22:01.457162 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:01.456829 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" podUID="ec6581e2-a031-44ce-860f-de3dd9c45642" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 05 17:22:03.158265 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.158242 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:22:03.235128 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.235040 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec6581e2-a031-44ce-860f-de3dd9c45642-istiod-ca-cert\") pod \"ec6581e2-a031-44ce-860f-de3dd9c45642\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " Aug 05 17:22:03.235128 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.235092 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwbzv\" (UniqueName: \"kubernetes.io/projected/ec6581e2-a031-44ce-860f-de3dd9c45642-kube-api-access-nwbzv\") pod \"ec6581e2-a031-44ce-860f-de3dd9c45642\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " Aug 05 17:22:03.235128 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.235121 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-workload-certs\") pod \"ec6581e2-a031-44ce-860f-de3dd9c45642\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " Aug 05 17:22:03.235450 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.235150 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-istio-envoy\") pod \"ec6581e2-a031-44ce-860f-de3dd9c45642\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " Aug 05 17:22:03.235450 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.235173 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-workload-socket\") pod \"ec6581e2-a031-44ce-860f-de3dd9c45642\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " Aug 05 17:22:03.235450 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.235227 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-credential-socket\") pod \"ec6581e2-a031-44ce-860f-de3dd9c45642\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " Aug 05 17:22:03.235450 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.235270 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec6581e2-a031-44ce-860f-de3dd9c45642-istio-token\") pod \"ec6581e2-a031-44ce-860f-de3dd9c45642\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " Aug 05 17:22:03.235450 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.235299 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-istio-data\") pod \"ec6581e2-a031-44ce-860f-de3dd9c45642\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " Aug 05 17:22:03.235450 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.235339 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec6581e2-a031-44ce-860f-de3dd9c45642-istio-podinfo\") pod \"ec6581e2-a031-44ce-860f-de3dd9c45642\" (UID: \"ec6581e2-a031-44ce-860f-de3dd9c45642\") " Aug 05 17:22:03.235744 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.235512 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ec6581e2-a031-44ce-860f-de3dd9c45642" (UID: "ec6581e2-a031-44ce-860f-de3dd9c45642"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:22:03.235744 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.235527 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ec6581e2-a031-44ce-860f-de3dd9c45642" (UID: "ec6581e2-a031-44ce-860f-de3dd9c45642"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:22:03.235744 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.235540 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ec6581e2-a031-44ce-860f-de3dd9c45642" (UID: "ec6581e2-a031-44ce-860f-de3dd9c45642"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:22:03.235744 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.235548 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec6581e2-a031-44ce-860f-de3dd9c45642-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ec6581e2-a031-44ce-860f-de3dd9c45642" (UID: "ec6581e2-a031-44ce-860f-de3dd9c45642"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:22:03.235955 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.235746 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-workload-certs\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:22:03.235955 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.235766 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-workload-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:22:03.235955 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.235779 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-credential-socket\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:22:03.235955 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.235792 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec6581e2-a031-44ce-860f-de3dd9c45642-istiod-ca-cert\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:22:03.235955 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.235748 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-istio-data" (OuterVolumeSpecName: "istio-data") pod "ec6581e2-a031-44ce-860f-de3dd9c45642" (UID: "ec6581e2-a031-44ce-860f-de3dd9c45642"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:22:03.237488 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.237448 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec6581e2-a031-44ce-860f-de3dd9c45642-kube-api-access-nwbzv" (OuterVolumeSpecName: "kube-api-access-nwbzv") pod "ec6581e2-a031-44ce-860f-de3dd9c45642" (UID: "ec6581e2-a031-44ce-860f-de3dd9c45642"). InnerVolumeSpecName "kube-api-access-nwbzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:22:03.237624 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.237514 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec6581e2-a031-44ce-860f-de3dd9c45642-istio-token" (OuterVolumeSpecName: "istio-token") pod "ec6581e2-a031-44ce-860f-de3dd9c45642" (UID: "ec6581e2-a031-44ce-860f-de3dd9c45642"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:22:03.237697 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.237621 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec6581e2-a031-44ce-860f-de3dd9c45642-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ec6581e2-a031-44ce-860f-de3dd9c45642" (UID: "ec6581e2-a031-44ce-860f-de3dd9c45642"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 17:22:03.237756 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.237716 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ec6581e2-a031-44ce-860f-de3dd9c45642" (UID: "ec6581e2-a031-44ce-860f-de3dd9c45642"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:22:03.302676 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.302638 2573 generic.go:356] "Generic (PLEG): container finished" podID="ec6581e2-a031-44ce-860f-de3dd9c45642" containerID="865cf0e0ff280ebc3aa84057834b58ceef0cedc88fe538e8b7f535c4389d8004" exitCode=0 Aug 05 17:22:03.302853 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.302767 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" Aug 05 17:22:03.304314 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.304289 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" event={"ID":"ec6581e2-a031-44ce-860f-de3dd9c45642","Type":"ContainerDied","Data":"865cf0e0ff280ebc3aa84057834b58ceef0cedc88fe538e8b7f535c4389d8004"} Aug 05 17:22:03.304430 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.304333 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp" event={"ID":"ec6581e2-a031-44ce-860f-de3dd9c45642","Type":"ContainerDied","Data":"63cf5dbdf8dcaa8647b487a10ce9a03097e71a66145cfb431f1987d166e69069"} Aug 05 17:22:03.304430 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.304357 2573 scope.go:117] "RemoveContainer" containerID="865cf0e0ff280ebc3aa84057834b58ceef0cedc88fe538e8b7f535c4389d8004" Aug 05 17:22:03.313347 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.313322 2573 scope.go:117] "RemoveContainer" containerID="10e6e9621112a862b37303989f0ceea93dd4e51ce249dbff3d7aa172fe9f5774" Aug 05 17:22:03.320970 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.320955 2573 scope.go:117] "RemoveContainer" containerID="865cf0e0ff280ebc3aa84057834b58ceef0cedc88fe538e8b7f535c4389d8004" Aug 05 17:22:03.321210 ip-10-0-132-37 kubenswrapper[2573]: E0805 17:22:03.321192 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"865cf0e0ff280ebc3aa84057834b58ceef0cedc88fe538e8b7f535c4389d8004\": container with ID starting with 865cf0e0ff280ebc3aa84057834b58ceef0cedc88fe538e8b7f535c4389d8004 not found: ID does not exist" containerID="865cf0e0ff280ebc3aa84057834b58ceef0cedc88fe538e8b7f535c4389d8004" Aug 05 17:22:03.321253 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.321218 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"865cf0e0ff280ebc3aa84057834b58ceef0cedc88fe538e8b7f535c4389d8004"} err="failed to get container status \"865cf0e0ff280ebc3aa84057834b58ceef0cedc88fe538e8b7f535c4389d8004\": rpc error: code = NotFound desc = could not find container \"865cf0e0ff280ebc3aa84057834b58ceef0cedc88fe538e8b7f535c4389d8004\": container with ID starting with 865cf0e0ff280ebc3aa84057834b58ceef0cedc88fe538e8b7f535c4389d8004 not found: ID does not exist" Aug 05 17:22:03.321253 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.321234 2573 scope.go:117] "RemoveContainer" containerID="10e6e9621112a862b37303989f0ceea93dd4e51ce249dbff3d7aa172fe9f5774" Aug 05 17:22:03.321464 ip-10-0-132-37 kubenswrapper[2573]: E0805 17:22:03.321446 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10e6e9621112a862b37303989f0ceea93dd4e51ce249dbff3d7aa172fe9f5774\": container with ID starting with 10e6e9621112a862b37303989f0ceea93dd4e51ce249dbff3d7aa172fe9f5774 not found: ID does not exist" containerID="10e6e9621112a862b37303989f0ceea93dd4e51ce249dbff3d7aa172fe9f5774" Aug 05 17:22:03.321533 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.321472 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e6e9621112a862b37303989f0ceea93dd4e51ce249dbff3d7aa172fe9f5774"} err="failed to get container status \"10e6e9621112a862b37303989f0ceea93dd4e51ce249dbff3d7aa172fe9f5774\": rpc error: code = NotFound desc = could not find container \"10e6e9621112a862b37303989f0ceea93dd4e51ce249dbff3d7aa172fe9f5774\": container with ID starting with 10e6e9621112a862b37303989f0ceea93dd4e51ce249dbff3d7aa172fe9f5774 not found: ID does not exist" Aug 05 17:22:03.329237 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.329216 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp"] Aug 05 17:22:03.333718 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.333698 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-46e6bf-openshift-default-6b97c6bcfb-kdlrp"] Aug 05 17:22:03.336410 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.336394 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-istio-data\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:22:03.336463 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.336412 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec6581e2-a031-44ce-860f-de3dd9c45642-istio-podinfo\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:22:03.336463 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.336421 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwbzv\" (UniqueName: \"kubernetes.io/projected/ec6581e2-a031-44ce-860f-de3dd9c45642-kube-api-access-nwbzv\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:22:03.336463 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.336430 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec6581e2-a031-44ce-860f-de3dd9c45642-istio-envoy\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:22:03.336463 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:03.336440 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec6581e2-a031-44ce-860f-de3dd9c45642-istio-token\") on node \"ip-10-0-132-37.ec2.internal\" DevicePath \"\"" Aug 05 17:22:04.201391 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:04.201352 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9778877d8-chcjf_f2370e77-a0e4-4344-8ae8-eb4d6ed2f987/copy-manifests/0.log" Aug 05 17:22:04.208327 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:04.208301 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9778877d8-chcjf_f2370e77-a0e4-4344-8ae8-eb4d6ed2f987/manager/0.log" Aug 05 17:22:04.883808 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:04.883774 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29765805-2fgnf_d5a997cd-da4f-4c33-ab6a-46ceb176f0e4/cleanup/0.log" Aug 05 17:22:04.989515 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:04.989482 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29765820-6q747_b03af45e-ed59-439d-9145-230a33c7df66/cleanup/0.log" Aug 05 17:22:05.304784 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:05.304753 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec6581e2-a031-44ce-860f-de3dd9c45642" path="/var/lib/kubelet/pods/ec6581e2-a031-44ce-860f-de3dd9c45642/volumes" Aug 05 17:22:05.920695 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:05.920664 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6cd787c69b-588vp_7f30f2b9-98de-4d12-846e-c3511d31201c/authorino/0.log" Aug 05 17:22:06.366823 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:06.366793 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-thkmv_962f33af-7194-4984-9aed-12c1bd3cb1ab/manager/0.log" Aug 05 17:22:06.597858 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:06.597827 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-p6m27_07f3e42e-2c78-4ac1-8b67-dd747f0326d3/manager/0.log" Aug 05 17:22:06.936068 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:06.936040 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5_d5b2c4fa-00c7-4ee1-869e-756c25aad1b6/istio-proxy/0.log" Aug 05 17:22:07.384583 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:07.384549 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-x7llj_4c8f3579-b836-485b-923c-21f2ba37336a/istio-proxy/1.log" Aug 05 17:22:07.714265 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:07.714179 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-cbf86bfcb-8pv7r_773c2ff3-528d-4d42-a88c-03331f44791c/router/0.log" Aug 05 17:22:08.254677 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:08.254642 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-t9nfr_bc6e8e0c-7870-4fe3-8e21-17d7e1798b0d/main/0.log" Aug 05 17:22:08.583253 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:08.583218 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-zfrvt_a3cc148c-4a43-4867-b50a-60e042c781bf/main/0.log" Aug 05 17:22:14.096245 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096208 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sjxd8/must-gather-59rcd"] Aug 05 17:22:14.096674 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096529 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2405d0d6-c0f2-4c17-bdfc-48ede7762288" containerName="istio-proxy" Aug 05 17:22:14.096674 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096540 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2405d0d6-c0f2-4c17-bdfc-48ede7762288" containerName="istio-proxy" Aug 05 17:22:14.096674 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096551 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec6581e2-a031-44ce-860f-de3dd9c45642" containerName="istio-proxy" Aug 05 17:22:14.096674 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096556 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec6581e2-a031-44ce-860f-de3dd9c45642" containerName="istio-proxy" Aug 05 17:22:14.096674 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096566 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="265e37c0-bc0b-421c-bbfa-2e7c74d7d731" containerName="istio-proxy" Aug 05 17:22:14.096674 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096571 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="265e37c0-bc0b-421c-bbfa-2e7c74d7d731" containerName="istio-proxy" Aug 05 17:22:14.096674 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096577 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ffa5a86-0a93-4fc8-83df-1f0f368f86e2" containerName="istio-proxy" Aug 05 17:22:14.096674 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096582 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ffa5a86-0a93-4fc8-83df-1f0f368f86e2" containerName="istio-proxy" Aug 05 17:22:14.096674 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096588 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="808215e7-6fbd-4dad-9b36-3e3cf05eec13" containerName="istio-proxy" Aug 05 17:22:14.096674 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096594 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="808215e7-6fbd-4dad-9b36-3e3cf05eec13" containerName="istio-proxy" Aug 05 17:22:14.096674 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096604 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="265e37c0-bc0b-421c-bbfa-2e7c74d7d731" containerName="istio-proxy" Aug 05 17:22:14.096674 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096609 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="265e37c0-bc0b-421c-bbfa-2e7c74d7d731" containerName="istio-proxy" Aug 05 17:22:14.096674 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096617 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec6581e2-a031-44ce-860f-de3dd9c45642" containerName="istio-proxy" Aug 05 17:22:14.096674 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096622 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec6581e2-a031-44ce-860f-de3dd9c45642" containerName="istio-proxy" Aug 05 17:22:14.096674 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096633 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="808215e7-6fbd-4dad-9b36-3e3cf05eec13" containerName="istio-proxy" Aug 05 17:22:14.096674 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096638 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="808215e7-6fbd-4dad-9b36-3e3cf05eec13" containerName="istio-proxy" Aug 05 17:22:14.096674 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096643 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ffa5a86-0a93-4fc8-83df-1f0f368f86e2" containerName="istio-proxy" Aug 05 17:22:14.096674 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096648 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ffa5a86-0a93-4fc8-83df-1f0f368f86e2" containerName="istio-proxy" Aug 05 17:22:14.097216 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096692 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2405d0d6-c0f2-4c17-bdfc-48ede7762288" containerName="istio-proxy" Aug 05 17:22:14.097216 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096699 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="808215e7-6fbd-4dad-9b36-3e3cf05eec13" containerName="istio-proxy" Aug 05 17:22:14.097216 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096706 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="265e37c0-bc0b-421c-bbfa-2e7c74d7d731" containerName="istio-proxy" Aug 05 17:22:14.097216 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096711 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec6581e2-a031-44ce-860f-de3dd9c45642" containerName="istio-proxy" Aug 05 17:22:14.097216 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096717 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec6581e2-a031-44ce-860f-de3dd9c45642" containerName="istio-proxy" Aug 05 17:22:14.097216 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096722 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ffa5a86-0a93-4fc8-83df-1f0f368f86e2" containerName="istio-proxy" Aug 05 17:22:14.097216 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096729 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ffa5a86-0a93-4fc8-83df-1f0f368f86e2" containerName="istio-proxy" Aug 05 17:22:14.097216 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096776 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2405d0d6-c0f2-4c17-bdfc-48ede7762288" containerName="istio-proxy" Aug 05 17:22:14.097216 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096782 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2405d0d6-c0f2-4c17-bdfc-48ede7762288" containerName="istio-proxy" Aug 05 17:22:14.097216 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096828 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2405d0d6-c0f2-4c17-bdfc-48ede7762288" containerName="istio-proxy" Aug 05 17:22:14.097216 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096833 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="808215e7-6fbd-4dad-9b36-3e3cf05eec13" containerName="istio-proxy" Aug 05 17:22:14.097216 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.096839 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="265e37c0-bc0b-421c-bbfa-2e7c74d7d731" containerName="istio-proxy" Aug 05 17:22:14.099566 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.099549 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sjxd8/must-gather-59rcd" Aug 05 17:22:14.102941 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.102920 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sjxd8\"/\"openshift-service-ca.crt\"" Aug 05 17:22:14.102941 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.102939 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sjxd8\"/\"kube-root-ca.crt\"" Aug 05 17:22:14.103101 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.102920 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sjxd8\"/\"default-dockercfg-mnzst\"" Aug 05 17:22:14.114091 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.114065 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sjxd8/must-gather-59rcd"] Aug 05 17:22:14.120917 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.120885 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:14.201712 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.201681 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" limit=1024 Aug 05 17:22:14.222767 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.222738 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0790b32a-3085-40a9-aadf-b07ecaff183c-must-gather-output\") pod \"must-gather-59rcd\" (UID: \"0790b32a-3085-40a9-aadf-b07ecaff183c\") " pod="openshift-must-gather-sjxd8/must-gather-59rcd" Aug 05 17:22:14.222855 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.222776 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t89xg\" (UniqueName: \"kubernetes.io/projected/0790b32a-3085-40a9-aadf-b07ecaff183c-kube-api-access-t89xg\") pod \"must-gather-59rcd\" (UID: \"0790b32a-3085-40a9-aadf-b07ecaff183c\") " pod="openshift-must-gather-sjxd8/must-gather-59rcd" Aug 05 17:22:14.323315 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.323279 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0790b32a-3085-40a9-aadf-b07ecaff183c-must-gather-output\") pod \"must-gather-59rcd\" (UID: \"0790b32a-3085-40a9-aadf-b07ecaff183c\") " pod="openshift-must-gather-sjxd8/must-gather-59rcd" Aug 05 17:22:14.323315 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.323314 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t89xg\" (UniqueName: \"kubernetes.io/projected/0790b32a-3085-40a9-aadf-b07ecaff183c-kube-api-access-t89xg\") pod \"must-gather-59rcd\" (UID: \"0790b32a-3085-40a9-aadf-b07ecaff183c\") " pod="openshift-must-gather-sjxd8/must-gather-59rcd" Aug 05 17:22:14.323708 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.323686 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0790b32a-3085-40a9-aadf-b07ecaff183c-must-gather-output\") pod \"must-gather-59rcd\" (UID: \"0790b32a-3085-40a9-aadf-b07ecaff183c\") " pod="openshift-must-gather-sjxd8/must-gather-59rcd" Aug 05 17:22:14.333017 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.332991 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t89xg\" (UniqueName: \"kubernetes.io/projected/0790b32a-3085-40a9-aadf-b07ecaff183c-kube-api-access-t89xg\") pod \"must-gather-59rcd\" (UID: \"0790b32a-3085-40a9-aadf-b07ecaff183c\") " pod="openshift-must-gather-sjxd8/must-gather-59rcd" Aug 05 17:22:14.409062 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.408975 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sjxd8/must-gather-59rcd" Aug 05 17:22:14.546273 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:14.546209 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sjxd8/must-gather-59rcd"] Aug 05 17:22:14.549064 ip-10-0-132-37 kubenswrapper[2573]: W0805 17:22:14.549037 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0790b32a_3085_40a9_aadf_b07ecaff183c.slice/crio-10bbfb7f81289abe1d72592a4ffbb392ca585691350f488aaff01ccc41014ecf WatchSource:0}: Error finding container 10bbfb7f81289abe1d72592a4ffbb392ca585691350f488aaff01ccc41014ecf: Status 404 returned error can't find the container with id 10bbfb7f81289abe1d72592a4ffbb392ca585691350f488aaff01ccc41014ecf Aug 05 17:22:15.362982 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:15.362893 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sjxd8/must-gather-59rcd" event={"ID":"0790b32a-3085-40a9-aadf-b07ecaff183c","Type":"ContainerStarted","Data":"163ba5f861cb93139d30dda73ad3be463c2677e10522ead99a44ea1b04beb79f"} Aug 05 17:22:15.362982 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:15.362950 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sjxd8/must-gather-59rcd" event={"ID":"0790b32a-3085-40a9-aadf-b07ecaff183c","Type":"ContainerStarted","Data":"10bbfb7f81289abe1d72592a4ffbb392ca585691350f488aaff01ccc41014ecf"} Aug 05 17:22:16.371102 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:16.371067 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sjxd8/must-gather-59rcd" event={"ID":"0790b32a-3085-40a9-aadf-b07ecaff183c","Type":"ContainerStarted","Data":"23e3c773e42b5c66aef4d01e6646fcbf0993bdd31901487965643b964aab2815"} Aug 05 17:22:17.020628 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:17.020602 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qjdbg_39d233df-9262-45b2-8065-0d44a7ee9e12/global-pull-secret-syncer/0.log" Aug 05 17:22:17.165730 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:17.165690 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kpd4q_333a43c6-e0d3-42ab-a596-3670a69c8d5d/konnectivity-agent/0.log" Aug 05 17:22:17.368939 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:17.368861 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-37.ec2.internal_14d058665423739b2594851afe44a8fc/haproxy/0.log" Aug 05 17:22:22.138173 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:22.137768 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6cd787c69b-588vp_7f30f2b9-98de-4d12-846e-c3511d31201c/authorino/0.log" Aug 05 17:22:22.390448 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:22.388146 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-thkmv_962f33af-7194-4984-9aed-12c1bd3cb1ab/manager/0.log" Aug 05 17:22:22.550414 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:22.549874 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-p6m27_07f3e42e-2c78-4ac1-8b67-dd747f0326d3/manager/0.log" Aug 05 17:22:24.431702 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:24.431621 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ffzbh_44168e30-8669-4130-bfe0-f64e6938aae4/node-exporter/0.log" Aug 05 17:22:24.458481 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:24.458454 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ffzbh_44168e30-8669-4130-bfe0-f64e6938aae4/kube-rbac-proxy/0.log" Aug 05 17:22:24.486857 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:24.486828 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ffzbh_44168e30-8669-4130-bfe0-f64e6938aae4/init-textfile/0.log" Aug 05 17:22:27.025219 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:27.025146 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sjxd8/must-gather-59rcd" podStartSLOduration=12.329796443 podStartE2EDuration="13.02511698s" totalImagesPullingTime="695.320537ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:22:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:22:16.390627376 +0000 UTC m=+3651.547473278" watchObservedRunningTime="2026-08-05 17:22:27.02511698 +0000 UTC m=+3662.181962882" Aug 05 17:22:27.025862 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:27.025832 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sjxd8/perf-node-gather-daemonset-dh2gw"] Aug 05 17:22:27.032263 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:27.032234 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sjxd8/perf-node-gather-daemonset-dh2gw" Aug 05 17:22:27.046393 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:27.044429 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sjxd8/perf-node-gather-daemonset-dh2gw"] Aug 05 17:22:27.154583 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:27.154545 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/830f1774-1e8b-43e6-9aa2-a2a2b3601cb6-lib-modules\") pod \"perf-node-gather-daemonset-dh2gw\" (UID: \"830f1774-1e8b-43e6-9aa2-a2a2b3601cb6\") " pod="openshift-must-gather-sjxd8/perf-node-gather-daemonset-dh2gw" Aug 05 17:22:27.154919 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:27.154894 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/830f1774-1e8b-43e6-9aa2-a2a2b3601cb6-podres\") pod \"perf-node-gather-daemonset-dh2gw\" (UID: \"830f1774-1e8b-43e6-9aa2-a2a2b3601cb6\") " pod="openshift-must-gather-sjxd8/perf-node-gather-daemonset-dh2gw" Aug 05 17:22:27.155047 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:27.155026 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/830f1774-1e8b-43e6-9aa2-a2a2b3601cb6-proc\") pod \"perf-node-gather-daemonset-dh2gw\" (UID: \"830f1774-1e8b-43e6-9aa2-a2a2b3601cb6\") " pod="openshift-must-gather-sjxd8/perf-node-gather-daemonset-dh2gw" Aug 05 17:22:27.155150 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:27.155126 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg4g7\" (UniqueName: \"kubernetes.io/projected/830f1774-1e8b-43e6-9aa2-a2a2b3601cb6-kube-api-access-bg4g7\") pod \"perf-node-gather-daemonset-dh2gw\" (UID: \"830f1774-1e8b-43e6-9aa2-a2a2b3601cb6\") " pod="openshift-must-gather-sjxd8/perf-node-gather-daemonset-dh2gw" Aug 05 17:22:27.155215 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:27.155178 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/830f1774-1e8b-43e6-9aa2-a2a2b3601cb6-sys\") pod \"perf-node-gather-daemonset-dh2gw\" (UID: \"830f1774-1e8b-43e6-9aa2-a2a2b3601cb6\") " pod="openshift-must-gather-sjxd8/perf-node-gather-daemonset-dh2gw" Aug 05 17:22:27.256369 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:27.256318 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/830f1774-1e8b-43e6-9aa2-a2a2b3601cb6-proc\") pod \"perf-node-gather-daemonset-dh2gw\" (UID: \"830f1774-1e8b-43e6-9aa2-a2a2b3601cb6\") " pod="openshift-must-gather-sjxd8/perf-node-gather-daemonset-dh2gw" Aug 05 17:22:27.256594 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:27.256472 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg4g7\" (UniqueName: \"kubernetes.io/projected/830f1774-1e8b-43e6-9aa2-a2a2b3601cb6-kube-api-access-bg4g7\") pod \"perf-node-gather-daemonset-dh2gw\" (UID: \"830f1774-1e8b-43e6-9aa2-a2a2b3601cb6\") " pod="openshift-must-gather-sjxd8/perf-node-gather-daemonset-dh2gw" Aug 05 17:22:27.256594 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:27.256519 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/830f1774-1e8b-43e6-9aa2-a2a2b3601cb6-sys\") pod \"perf-node-gather-daemonset-dh2gw\" (UID: \"830f1774-1e8b-43e6-9aa2-a2a2b3601cb6\") " pod="openshift-must-gather-sjxd8/perf-node-gather-daemonset-dh2gw" Aug 05 17:22:27.256594 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:27.256585 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/830f1774-1e8b-43e6-9aa2-a2a2b3601cb6-lib-modules\") pod \"perf-node-gather-daemonset-dh2gw\" (UID: \"830f1774-1e8b-43e6-9aa2-a2a2b3601cb6\") " pod="openshift-must-gather-sjxd8/perf-node-gather-daemonset-dh2gw" Aug 05 17:22:27.256775 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:27.256641 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/830f1774-1e8b-43e6-9aa2-a2a2b3601cb6-podres\") pod \"perf-node-gather-daemonset-dh2gw\" (UID: \"830f1774-1e8b-43e6-9aa2-a2a2b3601cb6\") " pod="openshift-must-gather-sjxd8/perf-node-gather-daemonset-dh2gw" Aug 05 17:22:27.256840 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:27.256825 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/830f1774-1e8b-43e6-9aa2-a2a2b3601cb6-podres\") pod \"perf-node-gather-daemonset-dh2gw\" (UID: \"830f1774-1e8b-43e6-9aa2-a2a2b3601cb6\") " pod="openshift-must-gather-sjxd8/perf-node-gather-daemonset-dh2gw" Aug 05 17:22:27.256914 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:27.256895 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/830f1774-1e8b-43e6-9aa2-a2a2b3601cb6-proc\") pod \"perf-node-gather-daemonset-dh2gw\" (UID: \"830f1774-1e8b-43e6-9aa2-a2a2b3601cb6\") " pod="openshift-must-gather-sjxd8/perf-node-gather-daemonset-dh2gw" Aug 05 17:22:27.257181 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:27.257152 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/830f1774-1e8b-43e6-9aa2-a2a2b3601cb6-sys\") pod \"perf-node-gather-daemonset-dh2gw\" (UID: \"830f1774-1e8b-43e6-9aa2-a2a2b3601cb6\") " pod="openshift-must-gather-sjxd8/perf-node-gather-daemonset-dh2gw" Aug 05 17:22:27.257248 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:27.257194 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/830f1774-1e8b-43e6-9aa2-a2a2b3601cb6-lib-modules\") pod \"perf-node-gather-daemonset-dh2gw\" (UID: \"830f1774-1e8b-43e6-9aa2-a2a2b3601cb6\") " pod="openshift-must-gather-sjxd8/perf-node-gather-daemonset-dh2gw" Aug 05 17:22:27.270322 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:27.270283 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg4g7\" (UniqueName: \"kubernetes.io/projected/830f1774-1e8b-43e6-9aa2-a2a2b3601cb6-kube-api-access-bg4g7\") pod \"perf-node-gather-daemonset-dh2gw\" (UID: \"830f1774-1e8b-43e6-9aa2-a2a2b3601cb6\") " pod="openshift-must-gather-sjxd8/perf-node-gather-daemonset-dh2gw" Aug 05 17:22:27.348244 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:27.348157 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sjxd8/perf-node-gather-daemonset-dh2gw" Aug 05 17:22:27.541456 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:27.540937 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sjxd8/perf-node-gather-daemonset-dh2gw"] Aug 05 17:22:27.543829 ip-10-0-132-37 kubenswrapper[2573]: W0805 17:22:27.543775 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod830f1774_1e8b_43e6_9aa2_a2a2b3601cb6.slice/crio-def7f8e4c44a649ffb7bfa52c3f619d92b10db15d2930749b81fab26abb43b0d WatchSource:0}: Error finding container def7f8e4c44a649ffb7bfa52c3f619d92b10db15d2930749b81fab26abb43b0d: Status 404 returned error can't find the container with id def7f8e4c44a649ffb7bfa52c3f619d92b10db15d2930749b81fab26abb43b0d Aug 05 17:22:28.461174 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:28.460233 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sjxd8/perf-node-gather-daemonset-dh2gw" event={"ID":"830f1774-1e8b-43e6-9aa2-a2a2b3601cb6","Type":"ContainerStarted","Data":"65021231a193def92538561cf86925acab8f7454a2f544b7db84f52eb1d8adc4"} Aug 05 17:22:28.461174 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:28.460331 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sjxd8/perf-node-gather-daemonset-dh2gw" event={"ID":"830f1774-1e8b-43e6-9aa2-a2a2b3601cb6","Type":"ContainerStarted","Data":"def7f8e4c44a649ffb7bfa52c3f619d92b10db15d2930749b81fab26abb43b0d"} Aug 05 17:22:28.461174 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:28.460615 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sjxd8/perf-node-gather-daemonset-dh2gw" Aug 05 17:22:28.483452 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:28.483398 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sjxd8/perf-node-gather-daemonset-dh2gw" podStartSLOduration=1.483365617 podStartE2EDuration="1.483365617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:22:28.480483419 +0000 UTC m=+3663.637329322" watchObservedRunningTime="2026-08-05 17:22:28.483365617 +0000 UTC m=+3663.640211554" Aug 05 17:22:28.715825 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:28.715727 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5b8776669-8vzp6_0e863683-b5a0-47e6-9d12-4a8539c943f7/volume-data-source-validator/0.log" Aug 05 17:22:29.857042 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:29.857010 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w8666_89e8400b-bfeb-44c4-8302-59ab47d1b4cf/dns/0.log" Aug 05 17:22:29.880184 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:29.880154 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w8666_89e8400b-bfeb-44c4-8302-59ab47d1b4cf/kube-rbac-proxy/0.log" Aug 05 17:22:29.963631 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:29.963600 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vbh4x_fe314431-1429-4319-b3a0-0c7287575cef/dns-node-resolver/0.log" Aug 05 17:22:30.529391 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:30.529337 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bmvw9_f2a0163b-3977-4a52-8c23-6139f01a9b2d/node-ca/0.log" Aug 05 17:22:31.480420 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:31.480388 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5_d5b2c4fa-00c7-4ee1-869e-756c25aad1b6/istio-proxy/0.log" Aug 05 17:22:31.749871 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:31.749791 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-x7llj_4c8f3579-b836-485b-923c-21f2ba37336a/istio-proxy/0.log" Aug 05 17:22:31.764523 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:31.764490 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-x7llj_4c8f3579-b836-485b-923c-21f2ba37336a/istio-proxy/1.log" Aug 05 17:22:31.860027 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:31.859990 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-cbf86bfcb-8pv7r_773c2ff3-528d-4d42-a88c-03331f44791c/router/0.log" Aug 05 17:22:32.679604 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:32.679572 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-477z7_76352a33-7d75-494f-9741-931a2cc97c40/serve-healthcheck-canary/0.log" Aug 05 17:22:33.413142 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:33.413108 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-srrzf_b42b84a0-641e-4650-a610-d2afa0f9dff1/kube-rbac-proxy/0.log" Aug 05 17:22:33.436336 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:33.436301 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-srrzf_b42b84a0-641e-4650-a610-d2afa0f9dff1/exporter/0.log" Aug 05 17:22:33.458869 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:33.458831 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-srrzf_b42b84a0-641e-4650-a610-d2afa0f9dff1/extractor/0.log" Aug 05 17:22:34.474640 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:34.474609 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sjxd8/perf-node-gather-daemonset-dh2gw" Aug 05 17:22:36.427343 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:36.427314 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9778877d8-chcjf_f2370e77-a0e4-4344-8ae8-eb4d6ed2f987/manager/0.log" Aug 05 17:22:36.449612 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:36.449585 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9778877d8-chcjf_f2370e77-a0e4-4344-8ae8-eb4d6ed2f987/copy-manifests/0.log" Aug 05 17:22:37.791757 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:37.791725 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5_d5b2c4fa-00c7-4ee1-869e-756c25aad1b6/istio-proxy/0.log" Aug 05 17:22:37.905609 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:37.905577 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6677bbd944-snbv2_0731c408-661d-4be4-8388-e1fc1cb301ab/manager/0.log" Aug 05 17:22:38.034187 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:38.034151 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-x7llj_4c8f3579-b836-485b-923c-21f2ba37336a/istio-proxy/0.log" Aug 05 17:22:38.052288 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:38.052203 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-x7llj_4c8f3579-b836-485b-923c-21f2ba37336a/istio-proxy/1.log" Aug 05 17:22:38.130643 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:38.130614 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-cbf86bfcb-8pv7r_773c2ff3-528d-4d42-a88c-03331f44791c/router/0.log" Aug 05 17:22:39.508854 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:39.508818 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-h2bl5_d5b2c4fa-00c7-4ee1-869e-756c25aad1b6/istio-proxy/0.log" Aug 05 17:22:39.749948 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:39.749909 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-x7llj_4c8f3579-b836-485b-923c-21f2ba37336a/istio-proxy/0.log" Aug 05 17:22:39.764573 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:39.764455 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-x7llj_4c8f3579-b836-485b-923c-21f2ba37336a/istio-proxy/1.log" Aug 05 17:22:39.827609 ip-10-0-132-37 kubenswrapper[2573]: I0805 17:22:39.827567 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-cbf86bfcb-8pv7r_773c2ff3-528d-4d42-a88c-03331f44791c/router/0.log"