Nov 02 21:05:50.709920 ip-10-0-128-89 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 21:05:50.709932 ip-10-0-128-89 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 21:05:50.709939 ip-10-0-128-89 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 21:05:50.710170 ip-10-0-128-89 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 21:06:00.808471 ip-10-0-128-89 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 21:06:00.808487 ip-10-0-128-89 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot a5ba59ff65e849259405137fc21e9914 -- Nov 02 21:07:45.232475 ip-10-0-128-89 systemd[1]: Starting Kubernetes Kubelet... Nov 02 21:07:45.599709 ip-10-0-128-89 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:07:45.599709 ip-10-0-128-89 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 21:07:45.599709 ip-10-0-128-89 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:07:45.599709 ip-10-0-128-89 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:07:45.604696 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604615 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:07:45.604696 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604695 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:07:45.604696 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604699 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:07:45.604790 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604703 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:07:45.604790 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604707 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:07:45.604790 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604710 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:07:45.604790 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604713 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:07:45.604790 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604723 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:07:45.604790 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604726 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:07:45.604790 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604728 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:07:45.604790 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604731 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:07:45.604790 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604733 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:07:45.604790 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604737 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:07:45.604790 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604739 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:07:45.604790 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604742 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:07:45.604790 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604745 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:07:45.604790 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604747 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:07:45.604790 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604750 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:07:45.604790 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604752 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:07:45.604790 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604755 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:07:45.604790 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604758 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:07:45.604790 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604761 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:07:45.604790 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604763 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:07:45.605376 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604766 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:07:45.605376 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604768 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:07:45.605376 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604771 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:07:45.605376 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604774 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:07:45.605376 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604776 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:07:45.605376 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604779 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:07:45.605376 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604781 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:07:45.605376 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604783 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:07:45.605376 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604786 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:07:45.605376 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604789 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:07:45.605376 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604791 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:07:45.605376 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604794 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:07:45.605376 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604797 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:07:45.605376 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604800 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:07:45.605376 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604803 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:07:45.605376 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604806 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:07:45.605376 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604808 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:07:45.605376 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604811 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:07:45.605376 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604814 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:07:45.605825 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604816 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:07:45.605825 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604818 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:07:45.605825 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604821 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:07:45.605825 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604823 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:07:45.605825 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604826 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:07:45.605825 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604829 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:07:45.605825 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604831 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:07:45.605825 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604834 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:07:45.605825 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604836 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:07:45.605825 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604838 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:07:45.605825 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604841 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:07:45.605825 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604843 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:07:45.605825 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604846 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:07:45.605825 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604849 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:07:45.605825 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604852 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:07:45.605825 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604855 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:07:45.605825 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604857 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:07:45.605825 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604860 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:07:45.605825 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604863 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:07:45.606288 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604865 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:07:45.606288 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604868 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:07:45.606288 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604871 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:07:45.606288 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604873 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:07:45.606288 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604876 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:07:45.606288 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604878 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:07:45.606288 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604881 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:07:45.606288 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604883 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:07:45.606288 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604886 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:07:45.606288 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604888 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:07:45.606288 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604891 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:07:45.606288 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604893 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:07:45.606288 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604896 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:07:45.606288 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604899 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:07:45.606288 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604902 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:07:45.606288 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604905 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:07:45.606288 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604909 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:07:45.606288 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604914 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:07:45.606288 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604917 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:07:45.606753 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604920 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:07:45.606753 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604923 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:07:45.606753 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604925 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:07:45.606753 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604928 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:07:45.606753 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604931 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:07:45.606753 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604933 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:07:45.606753 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604936 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:07:45.606753 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604939 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:07:45.606753 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604943 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:07:45.606753 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604947 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:07:45.606753 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604950 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:07:45.606753 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604953 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:07:45.606753 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604956 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:07:45.606753 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604959 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:07:45.606753 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604962 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:07:45.606753 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604965 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:07:45.606753 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604967 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:07:45.606753 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604970 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:07:45.606753 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604973 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:07:45.607257 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604975 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:07:45.607257 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604978 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:07:45.607257 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604980 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:07:45.607257 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604983 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:07:45.607257 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604987 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:07:45.607257 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604990 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:07:45.607257 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.604992 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:07:45.607257 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605811 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:07:45.607257 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605819 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:07:45.607257 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605823 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:07:45.607257 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605826 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:07:45.607257 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605829 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:07:45.607257 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605833 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:07:45.607257 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605836 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:07:45.607257 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605838 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:07:45.607257 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605841 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:07:45.607257 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605843 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:07:45.607257 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605846 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:07:45.607257 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605851 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:07:45.607714 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605853 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:07:45.607714 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605856 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:07:45.607714 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605859 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:07:45.607714 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605861 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:07:45.607714 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605864 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:07:45.607714 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605867 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:07:45.607714 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605869 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:07:45.607714 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605872 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:07:45.607714 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605875 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:07:45.607714 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605877 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:07:45.607714 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605880 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:07:45.607714 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605882 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:07:45.607714 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605885 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:07:45.607714 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605887 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:07:45.607714 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605890 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:07:45.607714 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605892 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:07:45.607714 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605895 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:07:45.607714 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605898 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:07:45.607714 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605901 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:07:45.608177 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605904 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:07:45.608177 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605906 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:07:45.608177 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605910 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:07:45.608177 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605913 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:07:45.608177 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605917 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:07:45.608177 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605921 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:07:45.608177 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605924 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:07:45.608177 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605928 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:07:45.608177 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605931 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:07:45.608177 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605933 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:07:45.608177 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605936 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:07:45.608177 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605938 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:07:45.608177 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605941 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:07:45.608177 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605944 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:07:45.608177 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605947 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:07:45.608177 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605949 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:07:45.608177 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605952 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:07:45.608177 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605954 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:07:45.608177 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605957 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:07:45.608177 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605960 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:07:45.608654 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605962 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:07:45.608654 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605965 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:07:45.608654 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605967 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:07:45.608654 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605970 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:07:45.608654 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605973 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:07:45.608654 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605977 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:07:45.608654 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605981 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:07:45.608654 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605985 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:07:45.608654 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605988 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:07:45.608654 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605990 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:07:45.608654 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605993 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:07:45.608654 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605996 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:07:45.608654 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.605999 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:07:45.608654 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606001 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:07:45.608654 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606004 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:07:45.608654 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606007 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:07:45.608654 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606010 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:07:45.608654 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606012 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:07:45.608654 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606015 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:07:45.609117 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606017 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:07:45.609117 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606020 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:07:45.609117 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606023 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:07:45.609117 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606025 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:07:45.609117 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606028 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:07:45.609117 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606030 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:07:45.609117 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606033 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:07:45.609117 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606035 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:07:45.609117 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606038 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:07:45.609117 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606040 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:07:45.609117 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606043 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:07:45.609117 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606045 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:07:45.609117 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606048 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:07:45.609117 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606050 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:07:45.609117 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606053 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:07:45.609117 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606055 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:07:45.609117 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606057 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:07:45.609117 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606060 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:07:45.609117 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606062 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:07:45.609117 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606065 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:07:45.609608 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606068 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:07:45.609608 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606070 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:07:45.609608 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606073 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:07:45.609608 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606076 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:07:45.609608 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606078 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:07:45.609608 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606081 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:07:45.609608 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606083 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:07:45.609608 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606086 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:07:45.609608 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606089 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:07:45.609608 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606092 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:07:45.609608 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606095 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:07:45.609608 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606098 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:07:45.609608 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606104 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:07:45.609608 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606107 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:07:45.609608 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606109 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:07:45.609608 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606112 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:07:45.609608 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606641 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:07:45.609608 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606647 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:07:45.609608 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606651 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:07:45.610062 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606654 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:07:45.610062 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606657 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:07:45.610062 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606659 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:07:45.610062 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606662 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:07:45.610062 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606665 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:07:45.610062 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606667 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:07:45.610062 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606670 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:07:45.610062 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606672 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:07:45.610062 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606675 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:07:45.610062 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606678 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:07:45.610062 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606681 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:07:45.610062 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606683 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:07:45.610062 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606685 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:07:45.610062 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606688 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:07:45.610062 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606692 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:07:45.610062 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606695 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:07:45.610062 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606699 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:07:45.610062 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606701 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:07:45.610062 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606704 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:07:45.610542 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606707 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:07:45.610542 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606709 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:07:45.610542 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606712 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:07:45.610542 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606716 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:07:45.610542 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606719 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:07:45.610542 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606721 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:07:45.610542 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606724 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:07:45.610542 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606727 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:07:45.610542 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606729 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:07:45.610542 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606732 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:07:45.610542 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606734 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:07:45.610542 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606737 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:07:45.610542 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606739 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:07:45.610542 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606742 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:07:45.610542 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606744 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:07:45.610542 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606747 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:07:45.610542 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606750 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:07:45.610542 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606754 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:07:45.610542 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606757 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:07:45.611002 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606760 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:07:45.611002 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606763 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:07:45.611002 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606766 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:07:45.611002 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606768 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:07:45.611002 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606771 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:07:45.611002 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606773 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:07:45.611002 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606776 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:07:45.611002 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606779 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:07:45.611002 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606781 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:07:45.611002 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606784 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:07:45.611002 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606789 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:07:45.611002 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606791 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:07:45.611002 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606794 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:07:45.611002 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606796 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:07:45.611002 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606799 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:07:45.611002 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606806 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:07:45.611002 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606808 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:07:45.611002 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606812 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:07:45.611002 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606815 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:07:45.611002 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606817 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:07:45.611500 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606820 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:07:45.611500 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606822 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:07:45.611500 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606825 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:07:45.611500 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606827 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:07:45.611500 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606830 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:07:45.611500 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606833 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:07:45.611500 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606835 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:07:45.611500 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606838 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:07:45.611500 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606840 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:07:45.611500 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606843 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:07:45.611500 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606846 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:07:45.611500 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606848 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:07:45.611500 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606853 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:07:45.611500 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606855 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:07:45.611500 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606858 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:07:45.611500 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606861 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:07:45.611500 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606863 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:07:45.611500 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606866 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:07:45.611500 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606868 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:07:45.611500 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606871 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:07:45.612001 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606873 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:07:45.612001 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606876 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:07:45.612001 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606878 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:07:45.612001 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606881 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:07:45.612001 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606884 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:07:45.612001 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606886 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:07:45.612001 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606889 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:07:45.612001 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606893 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:07:45.612001 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606896 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:07:45.612001 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606899 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:07:45.612001 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606902 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:07:45.612001 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606905 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:07:45.612001 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606907 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:07:45.612001 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606910 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:07:45.612001 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606912 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:07:45.612001 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606915 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:07:45.612001 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606918 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:07:45.612001 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606920 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:07:45.612001 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606923 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:07:45.612001 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606925 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:07:45.612475 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606928 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:07:45.612475 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606930 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:07:45.612475 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606933 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:07:45.612475 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606935 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:07:45.612475 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.606939 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:07:45.612475 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607745 2577 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 21:07:45.612475 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607756 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 21:07:45.612475 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607763 2577 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 21:07:45.612475 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607768 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 21:07:45.612475 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607772 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 21:07:45.612475 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607776 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 21:07:45.612475 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607780 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 21:07:45.612475 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607785 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 21:07:45.612475 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607788 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 21:07:45.612475 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607791 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 21:07:45.612475 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607795 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 21:07:45.612475 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607798 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 21:07:45.612475 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607801 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 21:07:45.612475 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607805 2577 flags.go:64] FLAG: --cgroup-root="" Nov 02 21:07:45.612475 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607811 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 21:07:45.612475 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607814 2577 flags.go:64] FLAG: --client-ca-file="" Nov 02 21:07:45.612475 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607817 2577 flags.go:64] FLAG: --cloud-provider="external" Nov 02 21:07:45.612475 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607820 2577 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 21:07:45.613024 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607826 2577 flags.go:64] FLAG: --cluster-domain="" Nov 02 21:07:45.613024 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607830 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 21:07:45.613024 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607833 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 21:07:45.613024 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607836 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 21:07:45.613024 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607841 2577 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 21:07:45.613024 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607846 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 21:07:45.613024 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607849 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 21:07:45.613024 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607852 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 21:07:45.613024 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607856 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 21:07:45.613024 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607859 2577 flags.go:64] FLAG: --contention-profiling="false" Nov 02 21:07:45.613024 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607862 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 21:07:45.613024 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607865 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 21:07:45.613024 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607868 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 21:07:45.613024 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607872 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 21:07:45.613024 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607877 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 21:07:45.613024 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607880 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 21:07:45.613024 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607883 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 21:07:45.613024 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607886 2577 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 21:07:45.613024 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607889 2577 flags.go:64] FLAG: --enable-server="true" Nov 02 21:07:45.613024 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607892 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 21:07:45.613024 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607896 2577 flags.go:64] FLAG: --event-burst="100" Nov 02 21:07:45.613024 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607900 2577 flags.go:64] FLAG: --event-qps="50" Nov 02 21:07:45.613024 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607903 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 21:07:45.613024 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607906 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 21:07:45.613647 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607909 2577 flags.go:64] FLAG: --eviction-hard="" Nov 02 21:07:45.613647 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607914 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 21:07:45.613647 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607917 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 21:07:45.613647 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607920 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 21:07:45.613647 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607925 2577 flags.go:64] FLAG: --eviction-soft="" Nov 02 21:07:45.613647 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607929 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 21:07:45.613647 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607932 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 21:07:45.613647 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607935 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 21:07:45.613647 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607938 2577 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 21:07:45.613647 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607942 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 21:07:45.613647 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607945 2577 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 21:07:45.613647 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607948 2577 flags.go:64] FLAG: --feature-gates="" Nov 02 21:07:45.613647 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607952 2577 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 21:07:45.613647 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607955 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 21:07:45.613647 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607958 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 21:07:45.613647 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607961 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 21:07:45.613647 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607965 2577 flags.go:64] FLAG: --healthz-port="10248" Nov 02 21:07:45.613647 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607968 2577 flags.go:64] FLAG: --help="false" Nov 02 21:07:45.613647 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607971 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:07:45.613647 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607974 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 21:07:45.613647 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607977 2577 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 21:07:45.613647 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607980 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 21:07:45.613647 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607984 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 21:07:45.614260 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607987 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 21:07:45.614260 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607990 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 21:07:45.614260 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607993 2577 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 21:07:45.614260 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607996 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 21:07:45.614260 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.607999 2577 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 21:07:45.614260 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608003 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 21:07:45.614260 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608006 2577 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 21:07:45.614260 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608008 2577 flags.go:64] FLAG: --kube-reserved="" Nov 02 21:07:45.614260 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608012 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 21:07:45.614260 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608015 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 21:07:45.614260 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608017 2577 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 21:07:45.614260 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608020 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 21:07:45.614260 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608023 2577 flags.go:64] FLAG: --lock-file="" Nov 02 21:07:45.614260 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608028 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 21:07:45.614260 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608031 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 21:07:45.614260 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608034 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 21:07:45.614260 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608039 2577 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 21:07:45.614260 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608043 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 21:07:45.614260 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608046 2577 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 21:07:45.614260 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608049 2577 flags.go:64] FLAG: --logging-format="text" Nov 02 21:07:45.614260 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608052 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 21:07:45.614260 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608055 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 21:07:45.614260 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608058 2577 flags.go:64] FLAG: --manifest-url="" Nov 02 21:07:45.614260 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608061 2577 flags.go:64] FLAG: --manifest-url-header="" Nov 02 21:07:45.614260 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608066 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 21:07:45.614877 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608069 2577 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 21:07:45.614877 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608073 2577 flags.go:64] FLAG: --max-pods="110" Nov 02 21:07:45.614877 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608076 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 21:07:45.614877 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608079 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 21:07:45.614877 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608083 2577 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 21:07:45.614877 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608085 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 21:07:45.614877 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608089 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 21:07:45.614877 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608092 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 21:07:45.614877 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608095 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 21:07:45.614877 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608101 2577 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 21:07:45.614877 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608105 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 21:07:45.614877 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608108 2577 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 21:07:45.614877 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608111 2577 flags.go:64] FLAG: --pod-cidr="" Nov 02 21:07:45.614877 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608114 2577 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 21:07:45.614877 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608117 2577 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 21:07:45.614877 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608120 2577 flags.go:64] FLAG: --pods-per-core="0" Nov 02 21:07:45.614877 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608123 2577 flags.go:64] FLAG: --port="10250" Nov 02 21:07:45.614877 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608137 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 21:07:45.614877 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608143 2577 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0ce983b9fe3d8064f" Nov 02 21:07:45.614877 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608148 2577 flags.go:64] FLAG: --qos-reserved="" Nov 02 21:07:45.614877 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608154 2577 flags.go:64] FLAG: --read-only-port="10255" Nov 02 21:07:45.614877 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608158 2577 flags.go:64] FLAG: --register-node="true" Nov 02 21:07:45.614877 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608161 2577 flags.go:64] FLAG: --register-with-taints="" Nov 02 21:07:45.614877 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608166 2577 flags.go:64] FLAG: --registry-burst="10" Nov 02 21:07:45.614877 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608169 2577 flags.go:64] FLAG: --registry-qps="5" Nov 02 21:07:45.614877 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608172 2577 flags.go:64] FLAG: --reserved-cpus="" Nov 02 21:07:45.615605 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608174 2577 flags.go:64] FLAG: --reserved-memory="" Nov 02 21:07:45.615605 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608179 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 21:07:45.615605 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608183 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 21:07:45.615605 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608186 2577 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 21:07:45.615605 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608189 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 21:07:45.615605 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608192 2577 flags.go:64] FLAG: --runonce="false" Nov 02 21:07:45.615605 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608195 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 21:07:45.615605 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608198 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 21:07:45.615605 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608202 2577 flags.go:64] FLAG: --seccomp-default="false" Nov 02 21:07:45.615605 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608205 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 21:07:45.615605 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608207 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 21:07:45.615605 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608211 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 21:07:45.615605 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608214 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 21:07:45.615605 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608217 2577 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 21:07:45.615605 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608220 2577 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 21:07:45.615605 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608223 2577 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 21:07:45.615605 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608226 2577 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 21:07:45.615605 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608230 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 21:07:45.615605 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608234 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 21:07:45.615605 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608237 2577 flags.go:64] FLAG: --system-cgroups="" Nov 02 21:07:45.615605 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608240 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 21:07:45.615605 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608245 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 21:07:45.615605 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608248 2577 flags.go:64] FLAG: --tls-cert-file="" Nov 02 21:07:45.615605 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608251 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 21:07:45.615605 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608256 2577 flags.go:64] FLAG: --tls-min-version="" Nov 02 21:07:45.616596 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608259 2577 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 21:07:45.616596 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608262 2577 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 21:07:45.616596 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608265 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 21:07:45.616596 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608268 2577 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 21:07:45.616596 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608271 2577 flags.go:64] FLAG: --v="2" Nov 02 21:07:45.616596 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608276 2577 flags.go:64] FLAG: --version="false" Nov 02 21:07:45.616596 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608280 2577 flags.go:64] FLAG: --vmodule="" Nov 02 21:07:45.616596 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608284 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 21:07:45.616596 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608287 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 21:07:45.616596 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608429 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:07:45.616596 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608432 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:07:45.616596 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608436 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:07:45.616596 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608439 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:07:45.616596 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608442 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:07:45.616596 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608444 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:07:45.616596 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608447 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:07:45.616596 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608449 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:07:45.616596 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608452 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:07:45.616596 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608455 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:07:45.616596 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608458 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:07:45.616596 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608461 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:07:45.616596 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608463 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:07:45.617269 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608466 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:07:45.617269 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608468 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:07:45.617269 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608472 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:07:45.617269 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608475 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:07:45.617269 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608477 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:07:45.617269 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608480 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:07:45.617269 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608482 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:07:45.617269 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608485 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:07:45.617269 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608487 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:07:45.617269 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608490 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:07:45.617269 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608492 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:07:45.617269 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608496 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:07:45.617269 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608500 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:07:45.617269 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608503 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:07:45.617269 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608505 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:07:45.617269 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608508 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:07:45.617269 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608511 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:07:45.617269 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608513 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:07:45.617269 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608516 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:07:45.617731 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608518 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:07:45.617731 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608522 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:07:45.617731 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608524 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:07:45.617731 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608527 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:07:45.617731 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608529 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:07:45.617731 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608532 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:07:45.617731 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608534 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:07:45.617731 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608537 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:07:45.617731 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608539 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:07:45.617731 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608542 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:07:45.617731 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608545 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:07:45.617731 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608547 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:07:45.617731 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608550 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:07:45.617731 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608554 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:07:45.617731 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608558 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:07:45.617731 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608562 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:07:45.617731 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608565 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:07:45.617731 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608568 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:07:45.617731 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608570 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:07:45.617731 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608573 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:07:45.618237 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608575 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:07:45.618237 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608578 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:07:45.618237 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608580 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:07:45.618237 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608583 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:07:45.618237 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608586 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:07:45.618237 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608588 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:07:45.618237 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608591 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:07:45.618237 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608594 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:07:45.618237 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608602 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:07:45.618237 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608606 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:07:45.618237 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608608 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:07:45.618237 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608611 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:07:45.618237 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608614 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:07:45.618237 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608616 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:07:45.618237 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608620 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:07:45.618237 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608623 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:07:45.618237 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608625 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:07:45.618237 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608628 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:07:45.618237 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608631 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:07:45.618237 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608633 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:07:45.618719 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608636 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:07:45.618719 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608638 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:07:45.618719 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608641 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:07:45.618719 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608643 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:07:45.618719 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608646 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:07:45.618719 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608648 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:07:45.618719 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608650 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:07:45.618719 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608654 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:07:45.618719 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608657 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:07:45.618719 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608660 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:07:45.618719 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608662 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:07:45.618719 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608665 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:07:45.618719 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608668 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:07:45.618719 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608671 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:07:45.618719 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608673 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:07:45.618719 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608676 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:07:45.618719 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608679 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:07:45.618719 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608681 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:07:45.618719 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608684 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:07:45.619232 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608686 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:07:45.619232 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608689 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:07:45.619232 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608692 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:07:45.619232 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608694 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:07:45.619232 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608697 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:07:45.619232 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608699 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:07:45.619232 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608702 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:07:45.619232 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608705 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:07:45.619232 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608709 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:07:45.619232 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608712 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:07:45.619232 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608715 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:07:45.619232 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608717 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:07:45.619232 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608720 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:07:45.619232 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608722 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:07:45.619232 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608725 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:07:45.619232 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.608754 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:07:45.619635 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608789 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:07:45.619635 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608793 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:07:45.619635 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608796 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:07:45.619635 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608798 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:07:45.619635 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608803 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:07:45.619635 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608805 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:07:45.619635 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608808 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:07:45.619635 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608811 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:07:45.619635 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608813 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:07:45.619635 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608816 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:07:45.619635 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608818 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:07:45.619635 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608822 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:07:45.619635 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608825 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:07:45.619635 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608827 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:07:45.619635 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608830 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:07:45.619635 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608833 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:07:45.619635 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608835 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:07:45.619635 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608838 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:07:45.619635 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608840 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:07:45.619635 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608844 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:07:45.620109 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608846 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:07:45.620109 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608849 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:07:45.620109 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608851 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:07:45.620109 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608854 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:07:45.620109 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608856 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:07:45.620109 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608860 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:07:45.620109 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608863 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:07:45.620109 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608865 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:07:45.620109 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608868 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:07:45.620109 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608870 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:07:45.620109 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608873 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:07:45.620109 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608875 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:07:45.620109 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608878 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:07:45.620109 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608880 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:07:45.620109 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608883 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:07:45.620109 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608886 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:07:45.620109 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608890 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:07:45.620109 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608892 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:07:45.620109 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608895 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:07:45.620573 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608897 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:07:45.620573 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608900 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:07:45.620573 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608902 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:07:45.620573 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608905 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:07:45.620573 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608909 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:07:45.620573 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608912 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:07:45.620573 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608915 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:07:45.620573 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608918 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:07:45.620573 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608920 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:07:45.620573 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608923 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:07:45.620573 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608925 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:07:45.620573 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608928 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:07:45.620573 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608930 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:07:45.620573 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608933 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:07:45.620573 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608935 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:07:45.620573 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608938 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:07:45.620573 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608940 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:07:45.620573 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608943 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:07:45.620573 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608946 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:07:45.620573 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608949 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:07:45.621059 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608951 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:07:45.621059 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608954 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:07:45.621059 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608957 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:07:45.621059 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608959 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:07:45.621059 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608962 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:07:45.621059 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608964 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:07:45.621059 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608967 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:07:45.621059 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608969 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:07:45.621059 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608972 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:07:45.621059 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608976 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:07:45.621059 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608979 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:07:45.621059 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608981 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:07:45.621059 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608983 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:07:45.621059 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608986 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:07:45.621059 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608988 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:07:45.621059 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608991 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:07:45.621059 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.608996 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:07:45.621059 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609000 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:07:45.621059 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609003 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:07:45.621557 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609006 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:07:45.621557 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609009 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:07:45.621557 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609011 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:07:45.621557 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609014 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:07:45.621557 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609017 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:07:45.621557 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609019 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:07:45.621557 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609022 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:07:45.621557 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609024 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:07:45.621557 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609027 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:07:45.621557 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609030 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:07:45.621557 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609032 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:07:45.621557 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609034 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:07:45.621557 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609037 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:07:45.621557 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609040 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:07:45.621557 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609043 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:07:45.621557 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609046 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:07:45.621557 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609048 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:07:45.621557 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609051 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:07:45.621557 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609055 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:07:45.622010 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609059 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:07:45.622010 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609062 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:07:45.622010 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609065 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:07:45.622010 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609068 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:07:45.622010 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609071 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:07:45.622010 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609074 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:07:45.622010 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609076 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:07:45.622010 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609079 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:07:45.622010 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.609082 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:07:45.622010 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.617489 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 21:07:45.622010 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.617505 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 21:07:45.622010 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.617520 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 21:07:45.622010 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.617525 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 21:07:45.622010 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617585 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:07:45.622010 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617590 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:07:45.622010 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617593 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:07:45.622010 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617596 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:07:45.622010 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617599 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:07:45.622010 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617602 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:07:45.622010 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617605 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:07:45.622576 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617608 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:07:45.622576 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617611 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:07:45.622576 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617613 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:07:45.622576 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617616 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:07:45.622576 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617619 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:07:45.622576 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617622 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:07:45.622576 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617624 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:07:45.622576 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617627 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:07:45.622576 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617630 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:07:45.622576 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617633 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:07:45.622576 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617636 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:07:45.622576 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617638 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:07:45.622576 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617641 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:07:45.622576 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617643 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:07:45.622576 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617646 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:07:45.622576 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617649 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:07:45.622576 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617652 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:07:45.622576 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617655 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:07:45.622576 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617657 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:07:45.623058 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617660 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:07:45.623058 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617668 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:07:45.623058 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617671 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:07:45.623058 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617674 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:07:45.623058 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617677 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:07:45.623058 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617679 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:07:45.623058 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617682 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:07:45.623058 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617685 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:07:45.623058 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617687 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:07:45.623058 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617691 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:07:45.623058 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617695 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:07:45.623058 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617699 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:07:45.623058 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617702 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:07:45.623058 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617705 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:07:45.623058 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617709 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:07:45.623058 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617712 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:07:45.623058 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617716 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:07:45.623058 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617719 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:07:45.623058 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617722 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:07:45.623531 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617725 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:07:45.623531 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617728 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:07:45.623531 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617731 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:07:45.623531 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617734 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:07:45.623531 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617737 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:07:45.623531 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617739 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:07:45.623531 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617742 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:07:45.623531 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617745 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:07:45.623531 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617747 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:07:45.623531 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617750 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:07:45.623531 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617753 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:07:45.623531 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617756 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:07:45.623531 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617758 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:07:45.623531 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617761 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:07:45.623531 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617764 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:07:45.623531 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617766 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:07:45.623531 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617769 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:07:45.623531 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617771 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:07:45.623531 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617774 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:07:45.623531 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617777 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:07:45.624008 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617779 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:07:45.624008 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617782 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:07:45.624008 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617784 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:07:45.624008 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617787 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:07:45.624008 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617789 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:07:45.624008 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617792 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:07:45.624008 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617794 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:07:45.624008 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617797 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:07:45.624008 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617799 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:07:45.624008 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617802 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:07:45.624008 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617804 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:07:45.624008 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617807 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:07:45.624008 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617809 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:07:45.624008 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617812 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:07:45.624008 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617815 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:07:45.624008 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617817 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:07:45.624008 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617820 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:07:45.624008 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617822 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:07:45.624008 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617825 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:07:45.624483 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617828 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:07:45.624483 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617831 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:07:45.624483 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617833 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:07:45.624483 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617836 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:07:45.624483 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617839 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:07:45.624483 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617841 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:07:45.624483 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617844 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:07:45.624483 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617847 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:07:45.624483 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617849 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:07:45.624483 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617852 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:07:45.624483 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617854 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:07:45.624483 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617857 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:07:45.624483 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617859 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:07:45.624483 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617862 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:07:45.624483 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617864 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:07:45.624483 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617867 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:07:45.624483 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617869 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:07:45.624483 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617872 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:07:45.624483 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617874 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:07:45.624483 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617877 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:07:45.625032 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617879 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:07:45.625032 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.617882 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:07:45.625032 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.617910 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:07:45.625032 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618077 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:07:45.625032 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618083 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:07:45.625032 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618086 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:07:45.625032 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618089 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:07:45.625032 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618092 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:07:45.625032 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618096 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:07:45.625032 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618098 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:07:45.625032 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618101 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:07:45.625032 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618103 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:07:45.625032 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618106 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:07:45.625032 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618109 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:07:45.625032 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618112 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:07:45.625032 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618114 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:07:45.625426 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618117 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:07:45.625426 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618120 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:07:45.625426 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618123 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:07:45.625426 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618125 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:07:45.625426 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618221 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:07:45.625426 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618226 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:07:45.625426 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618231 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:07:45.625426 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618235 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:07:45.625426 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618240 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:07:45.625426 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618244 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:07:45.625426 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618248 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:07:45.625426 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618252 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:07:45.625426 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618260 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:07:45.625426 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618267 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:07:45.625426 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618271 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:07:45.625426 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618275 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:07:45.625426 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618280 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:07:45.625426 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618285 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:07:45.625426 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618289 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:07:45.625865 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618293 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:07:45.625865 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618297 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:07:45.625865 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618301 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:07:45.625865 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618305 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:07:45.625865 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618309 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:07:45.625865 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618313 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:07:45.625865 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618318 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:07:45.625865 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618322 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:07:45.625865 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618325 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:07:45.625865 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618329 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:07:45.625865 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618335 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:07:45.625865 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618339 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:07:45.625865 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618343 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:07:45.625865 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618346 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:07:45.625865 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618351 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:07:45.625865 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618355 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:07:45.625865 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618359 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:07:45.625865 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618363 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:07:45.625865 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618368 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:07:45.625865 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618372 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:07:45.626469 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618377 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:07:45.626469 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618381 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:07:45.626469 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618384 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:07:45.626469 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618388 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:07:45.626469 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618392 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:07:45.626469 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618396 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:07:45.626469 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618399 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:07:45.626469 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618403 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:07:45.626469 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618407 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:07:45.626469 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618410 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:07:45.626469 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618414 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:07:45.626469 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618419 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:07:45.626469 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618423 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:07:45.626469 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618427 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:07:45.626469 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618431 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:07:45.626469 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618436 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:07:45.626469 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618441 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:07:45.626469 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618446 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:07:45.626469 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618449 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:07:45.626469 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618454 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:07:45.626955 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618459 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:07:45.626955 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618463 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:07:45.626955 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618467 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:07:45.626955 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618471 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:07:45.626955 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618475 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:07:45.626955 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618478 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:07:45.626955 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618482 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:07:45.626955 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618486 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:07:45.626955 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618490 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:07:45.626955 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618494 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:07:45.626955 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618498 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:07:45.626955 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618502 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:07:45.626955 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618506 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:07:45.626955 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618511 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:07:45.626955 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618515 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:07:45.626955 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618519 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:07:45.626955 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618524 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:07:45.626955 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618528 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:07:45.626955 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618533 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:07:45.627523 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618537 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:07:45.627523 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618541 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:07:45.627523 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618547 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:07:45.627523 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618554 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:07:45.627523 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618559 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:07:45.627523 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618564 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:07:45.627523 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618568 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:07:45.627523 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618573 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:07:45.627523 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618577 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:07:45.627523 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618582 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:07:45.627523 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618586 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:07:45.627523 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618590 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:07:45.627523 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618594 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:07:45.627523 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618599 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:07:45.627523 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618602 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:07:45.627523 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.618655 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:07:45.627959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618720 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:07:45.627959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618728 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:07:45.627959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618733 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:07:45.627959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618738 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:07:45.627959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618743 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:07:45.627959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618747 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:07:45.627959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618753 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:07:45.627959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618757 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:07:45.627959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618761 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:07:45.627959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618766 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:07:45.627959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618770 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:07:45.627959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618775 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:07:45.627959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618780 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:07:45.627959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618784 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:07:45.627959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618789 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:07:45.627959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618794 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:07:45.627959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618799 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:07:45.627959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618803 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:07:45.627959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618808 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:07:45.627959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618813 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:07:45.628476 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618817 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:07:45.628476 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618822 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:07:45.628476 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618827 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:07:45.628476 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618831 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:07:45.628476 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618836 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:07:45.628476 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618840 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:07:45.628476 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618844 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:07:45.628476 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618855 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:07:45.628476 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618863 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:07:45.628476 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618868 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:07:45.628476 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618875 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:07:45.628476 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618880 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:07:45.628476 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618884 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:07:45.628476 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618889 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:07:45.628476 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618893 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:07:45.628476 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618898 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:07:45.628476 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618902 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:07:45.628476 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618907 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:07:45.628476 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618912 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:07:45.628959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618916 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:07:45.628959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618921 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:07:45.628959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618925 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:07:45.628959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618930 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:07:45.628959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618935 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:07:45.628959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618939 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:07:45.628959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618944 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:07:45.628959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618948 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:07:45.628959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618953 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:07:45.628959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618958 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:07:45.628959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618963 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:07:45.628959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618967 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:07:45.628959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618970 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:07:45.628959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618974 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:07:45.628959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618978 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:07:45.628959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618981 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:07:45.628959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618985 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:07:45.628959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618989 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:07:45.628959 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618993 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:07:45.629458 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.618997 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:07:45.629458 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619003 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:07:45.629458 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619007 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:07:45.629458 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619011 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:07:45.629458 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619017 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:07:45.629458 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619022 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:07:45.629458 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619026 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:07:45.629458 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619032 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:07:45.629458 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619037 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:07:45.629458 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619041 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:07:45.629458 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619045 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:07:45.629458 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619049 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:07:45.629458 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619054 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:07:45.629458 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619059 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:07:45.629458 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619063 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:07:45.629458 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619067 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:07:45.629458 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619072 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:07:45.629458 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619077 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:07:45.629458 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619082 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:07:45.629458 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619086 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:07:45.629931 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619091 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:07:45.629931 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619096 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:07:45.629931 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619100 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:07:45.629931 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619104 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:07:45.629931 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619108 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:07:45.629931 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619112 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:07:45.629931 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619118 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:07:45.629931 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619122 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:07:45.629931 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619125 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:07:45.629931 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619145 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:07:45.629931 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619150 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:07:45.629931 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619155 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:07:45.629931 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619160 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:07:45.629931 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619167 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:07:45.629931 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619172 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:07:45.629931 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619175 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:07:45.629931 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619183 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:07:45.629931 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619189 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:07:45.629931 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619197 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:07:45.630397 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619202 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:07:45.630397 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619206 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:07:45.630397 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619211 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:07:45.630397 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619216 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:07:45.630397 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619222 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:07:45.630397 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619225 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:07:45.630397 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619228 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:07:45.630397 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619231 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:07:45.630397 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:45.619236 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:07:45.630397 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.619304 2577 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 21:07:45.630397 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.621500 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 21:07:45.630397 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.622450 2577 server.go:1014] "Starting client certificate rotation" Nov 02 21:07:45.630397 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.622544 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 21:07:45.630397 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.622583 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 21:07:45.630397 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.624709 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:07:45.630397 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.626115 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:07:45.637116 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.637099 2577 log.go:25] "Validated CRI v1 runtime API" Nov 02 21:07:45.640834 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.640819 2577 log.go:25] "Validated CRI v1 image API" Nov 02 21:07:45.641664 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.641630 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 21:07:45.645000 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.644982 2577 fs.go:135] Filesystem UUIDs: map[387dbd0c-85bc-41c1-b895-d36e9307ce67:/dev/nvme0n1p4 5f8ee96b-f322-4201-ba54-0cca4207be19:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 02 21:07:45.645050 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.645001 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 21:07:45.650146 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.650029 2577 manager.go:217] Machine: {Timestamp:2026-11-02 21:07:45.648668101 +0000 UTC m=+0.305742981 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099235 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23b5015d2620ed3d048a7909b84980 SystemUUID:ec23b501-5d26-20ed-3d04-8a7909b84980 BootID:a5ba59ff-65e8-4925-9405-137fc21e9914 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:d6:d3:1e:85 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:d6:d3:1e:85 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:83:24:39:e3:a3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 21:07:45.650229 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.650148 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 21:07:45.650263 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.650254 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 21:07:45.651519 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.651491 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 21:07:45.651652 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.651522 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-89.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 21:07:45.651700 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.651658 2577 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 21:07:45.651700 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.651666 2577 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 21:07:45.651700 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.651693 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 21:07:45.651772 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.651711 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 21:07:45.652935 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.652925 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 21:07:45.657466 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.657455 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 21:07:45.657589 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.657580 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 21:07:45.658446 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.658436 2577 kubelet.go:493] "Attempting to sync node with API server" Nov 02 21:07:45.658480 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.658452 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 21:07:45.658480 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.658462 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 21:07:45.658480 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.658471 2577 kubelet.go:405] "Adding apiserver pod source" Nov 02 21:07:45.658480 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.658479 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 21:07:45.658910 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.658891 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 21:07:45.662570 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.662546 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 21:07:45.669577 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.669558 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 21:07:45.670426 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.670412 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 21:07:45.670469 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.670444 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 21:07:45.670610 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.670600 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 21:07:45.670651 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.670616 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 21:07:45.670651 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.670623 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 21:07:45.670651 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.670628 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 21:07:45.670651 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.670634 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 21:07:45.670651 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.670640 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 21:07:45.671396 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.671387 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 21:07:45.671396 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.671397 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 21:07:45.671450 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.671404 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 21:07:45.671450 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.671410 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 21:07:45.671450 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.671419 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 21:07:45.671450 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.671428 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 21:07:45.672798 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.672788 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 21:07:45.672798 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.672798 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 21:07:45.676753 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.676737 2577 server.go:1263] "Started kubelet" Nov 02 21:07:45.676865 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.676830 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 21:07:45.676957 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.676854 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 21:07:45.677004 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.676984 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 21:07:45.677493 ip-10-0-128-89 systemd[1]: Started Kubernetes Kubelet. Nov 02 21:07:45.678455 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.678335 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 21:07:45.679700 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.679686 2577 server.go:317] "Adding debug handlers to kubelet server" Nov 02 21:07:45.683887 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:45.683866 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 21:07:45.687080 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.687062 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 21:07:45.687186 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.687103 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 21:07:45.687186 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:45.687156 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-89.us-west-2.compute.internal\" not found" Nov 02 21:07:45.687830 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.687815 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 21:07:45.687943 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.687927 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 21:07:45.688039 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.687815 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 21:07:45.688099 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.688073 2577 reconstruct.go:97] "Volume reconstruction finished" Nov 02 21:07:45.688099 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.688084 2577 reconciler.go:29] "Reconciler: start to sync state" Nov 02 21:07:45.688215 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.688203 2577 factory.go:55] Registering systemd factory Nov 02 21:07:45.688256 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.688237 2577 factory.go:223] Registration of the systemd container factory successfully Nov 02 21:07:45.688430 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.688417 2577 factory.go:153] Registering CRI-O factory Nov 02 21:07:45.688467 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.688431 2577 factory.go:223] Registration of the crio container factory successfully Nov 02 21:07:45.688495 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.688471 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 21:07:45.688495 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.688485 2577 factory.go:103] Registering Raw factory Nov 02 21:07:45.688495 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.688495 2577 manager.go:1199] Started watching for new ooms in manager Nov 02 21:07:45.689507 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.689491 2577 manager.go:319] Starting recovery of all containers Nov 02 21:07:45.701622 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.701596 2577 manager.go:324] Recovery completed Nov 02 21:07:45.704214 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:45.704110 2577 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Nov 02 21:07:45.707156 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.707125 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:07:45.709093 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.709073 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-89.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:07:45.709182 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.709112 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-89.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:07:45.709182 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.709126 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-89.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:07:45.709720 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.709707 2577 cpu_manager.go:226] "Starting" policy="none" Nov 02 21:07:45.709720 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.709717 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 21:07:45.709818 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.709720 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:07:45.709818 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.709737 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 21:07:45.710512 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.710496 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-89.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:07:45.710587 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.710527 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-89.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:07:45.710587 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.710540 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-89.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:07:45.711960 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.711948 2577 policy_none.go:50] "Start" Nov 02 21:07:45.712006 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.711965 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 21:07:45.712006 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.711983 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 21:07:45.713971 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.713959 2577 policy_none.go:44] "Start" Nov 02 21:07:45.775578 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.760827 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 21:07:45.775578 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.763540 2577 manager.go:341] "Starting Device Plugin manager" Nov 02 21:07:45.775578 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:45.763572 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 21:07:45.775578 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.763581 2577 server.go:87] "Starting device plugin registration server" Nov 02 21:07:45.775578 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.763850 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 21:07:45.775578 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.763865 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 21:07:45.775578 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.764011 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 21:07:45.775578 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.764032 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 21:07:45.775578 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.764097 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 21:07:45.775578 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.764104 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 21:07:45.775578 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:45.764925 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 21:07:45.775578 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:45.764978 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-89.us-west-2.compute.internal\" not found" Nov 02 21:07:45.775578 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.765576 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 21:07:45.775578 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.765598 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 21:07:45.775578 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.765621 2577 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 21:07:45.775578 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:45.765657 2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 21:07:45.814498 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.814459 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-89.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 21:07:45.814498 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:45.814460 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-89.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 21:07:45.815668 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:45.814461 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-89.us-west-2.compute.internal.18e4568178bc9667 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-89.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-89.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 21:07:45.676695143 +0000 UTC m=+0.333770024,LastTimestamp:2026-11-02 21:07:45.676695143 +0000 UTC m=+0.333770024,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-89.us-west-2.compute.internal,}" Nov 02 21:07:45.864167 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.864071 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:07:45.865127 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.865104 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-89.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:07:45.865239 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.865157 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-89.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:07:45.865239 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.865170 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-89.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:07:45.865239 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.865198 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:07:45.865736 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.865720 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-89.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal"] Nov 02 21:07:45.865785 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.865775 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:07:45.866495 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.866480 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-89.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:07:45.866576 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.866505 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-89.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:07:45.866576 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.866519 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-89.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:07:45.868477 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.868464 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:07:45.868633 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.868617 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:07:45.868695 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.868654 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:07:45.869275 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.869253 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-89.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:07:45.869369 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.869285 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-89.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:07:45.869369 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.869300 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-89.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:07:45.869369 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.869256 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-89.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:07:45.870118 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.870104 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-89.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:07:45.870234 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.870141 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-89.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:07:45.871074 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.871059 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:07:45.871171 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.871084 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:07:45.871763 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.871745 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-89.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:07:45.871832 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.871775 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-89.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:07:45.871832 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.871793 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-89.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:07:45.888853 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.888831 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/398d07e61029ddf8dea61a5829612cad-config\") pod \"kube-apiserver-proxy-ip-10-0-128-89.us-west-2.compute.internal\" (UID: \"398d07e61029ddf8dea61a5829612cad\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:07:45.888940 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.888854 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/30622891ad10d753b5b3452735f59810-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal\" (UID: \"30622891ad10d753b5b3452735f59810\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:07:45.888940 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.888892 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 21:07:45.889024 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.888946 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vhb4x" Nov 02 21:07:45.889298 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.888873 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30622891ad10d753b5b3452735f59810-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal\" (UID: \"30622891ad10d753b5b3452735f59810\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:07:45.897427 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:45.897410 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-89.us-west-2.compute.internal\" not found" node="ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:07:45.904205 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:45.904190 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-89.us-west-2.compute.internal\" not found" node="ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:07:45.964510 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.964490 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:07:45.966691 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.966675 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:07:45.989811 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.989782 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30622891ad10d753b5b3452735f59810-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal\" (UID: \"30622891ad10d753b5b3452735f59810\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:07:45.989897 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.989818 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/398d07e61029ddf8dea61a5829612cad-config\") pod \"kube-apiserver-proxy-ip-10-0-128-89.us-west-2.compute.internal\" (UID: \"398d07e61029ddf8dea61a5829612cad\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:07:45.989897 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.989841 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/30622891ad10d753b5b3452735f59810-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal\" (UID: \"30622891ad10d753b5b3452735f59810\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:07:45.989897 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.989873 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30622891ad10d753b5b3452735f59810-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal\" (UID: \"30622891ad10d753b5b3452735f59810\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:07:45.989897 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.989888 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/398d07e61029ddf8dea61a5829612cad-config\") pod \"kube-apiserver-proxy-ip-10-0-128-89.us-west-2.compute.internal\" (UID: \"398d07e61029ddf8dea61a5829612cad\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:07:45.990042 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:45.989874 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/30622891ad10d753b5b3452735f59810-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal\" (UID: \"30622891ad10d753b5b3452735f59810\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:07:46.060536 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.060501 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:07:46.061696 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.061677 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:07:46.061785 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.061700 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:07:46.061785 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:46.061706 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-89.us-west-2.compute.internal\": node \"ip-10-0-128-89.us-west-2.compute.internal\" not found" Nov 02 21:07:46.088043 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.088019 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:07:46.156623 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.156542 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:07:46.157454 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.157441 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:07:46.202126 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.202102 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:07:46.209180 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.209164 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:07:46.257094 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.257071 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:07:46.623189 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.623162 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 21:07:46.623624 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.623273 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 21:07:46.623624 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.623288 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 21:07:46.623624 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.623300 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 21:07:46.623624 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:46.623299 2577 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ab303f828c2034a3dad08796677c6e91-0df893518692fff8.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-128-89.us-west-2.compute.internal.18e456817aaaf0f3\": read tcp 10.0.128.89:40768->18.227.67.248:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-128-89.us-west-2.compute.internal.18e456817aaaf0f3 default 4838 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-89.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-89.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-89.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 21:07:45 +0000 UTC,LastTimestamp:2026-11-02 21:07:45.866493353 +0000 UTC m=+0.523568236,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-89.us-west-2.compute.internal,}" Nov 02 21:07:46.659965 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.659943 2577 apiserver.go:52] "Watching apiserver" Nov 02 21:07:46.687509 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.687447 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 21:07:46.894831 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.894773 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 21:02:45 +0000 UTC" deadline="2028-08-20 09:55:47.574877336 +0000 UTC" Nov 02 21:07:46.894831 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.894823 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15756h48m0.680059309s" Nov 02 21:07:46.912071 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.912038 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 21:07:46.912386 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.912368 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-z5hfn","openshift-dns/node-resolver-gxdgr","openshift-multus/multus-additional-cni-plugins-94zch","openshift-multus/network-metrics-daemon-s98xl","openshift-network-diagnostics/network-check-target-8c67p","kube-system/kube-apiserver-proxy-ip-10-0-128-89.us-west-2.compute.internal","openshift-image-registry/node-ca-77xn9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal","openshift-multus/multus-6blt8","openshift-network-operator/iptables-alerter-zpkht","openshift-ovn-kubernetes/ovnkube-node-pqnmg","kube-system/global-pull-secret-syncer-bllxh","kube-system/konnectivity-agent-mz5qm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n"] Nov 02 21:07:46.914569 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.914555 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:07:46.916372 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.916356 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zpkht" Nov 02 21:07:46.918276 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.918262 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:07:46.918337 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:46.918315 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s98xl" podUID="d45d9164-2ba4-400d-8b27-273d7f4f351d" Nov 02 21:07:46.920069 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.920052 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:46.921958 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.921944 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:07:46.922035 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:46.922002 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c67p" podUID="c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7" Nov 02 21:07:46.923862 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.923849 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-77xn9" Nov 02 21:07:46.925732 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.925720 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6blt8" Nov 02 21:07:46.927569 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.927554 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gxdgr" Nov 02 21:07:46.929488 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.929470 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:46.931531 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.931518 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:07:46.931584 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:46.931562 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bllxh" podUID="87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1" Nov 02 21:07:46.933512 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.933497 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mz5qm" Nov 02 21:07:46.935565 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.935551 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:07:46.964417 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.964395 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 21:07:46.964520 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.964456 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 21:07:46.965510 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.964837 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 21:07:46.965510 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.964877 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 21:07:46.965510 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.964894 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 21:07:46.966070 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.966047 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 21:07:46.966178 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.966156 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:07:46.967223 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.967206 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 21:07:46.967302 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.967285 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-894d2\"" Nov 02 21:07:46.967411 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.967399 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zzvdw\"" Nov 02 21:07:46.968682 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.968588 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fx9vz\"" Nov 02 21:07:46.968826 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.968797 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 21:07:46.968897 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.968882 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:07:46.972568 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.972554 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 21:07:46.972981 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.972969 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 21:07:46.972981 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.972976 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 21:07:46.973043 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.973023 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f8dkx\"" Nov 02 21:07:46.988799 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.988667 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 21:07:46.995462 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.995438 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-host\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:46.995535 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.995471 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq5t9\" (UniqueName: \"kubernetes.io/projected/42b83ba5-5e98-459f-ad4f-6a94c90212be-kube-api-access-lq5t9\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:07:46.995535 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.995491 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-kubernetes\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:46.995636 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.995536 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-var-lib-cni-bin\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:46.995636 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.995592 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-run-multus-certs\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:46.995636 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.995625 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bff8d097-26a3-4621-ac01-c3b00ed23e06-ovnkube-script-lib\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:46.995777 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.995669 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:07:46.995777 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.995717 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-sysctl-conf\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:46.995777 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.995766 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-tuned\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:46.995878 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.995783 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:07:46.995878 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.995797 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-run-openvswitch\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:46.995878 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.995811 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d992040e-0d1e-4227-befb-cfef1acd8aad-iptables-alerter-script\") pod \"iptables-alerter-zpkht\" (UID: \"d992040e-0d1e-4227-befb-cfef1acd8aad\") " pod="openshift-network-operator/iptables-alerter-zpkht" Nov 02 21:07:46.995878 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.995826 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txdkn\" (UniqueName: \"kubernetes.io/projected/d45d9164-2ba4-400d-8b27-273d7f4f351d-kube-api-access-txdkn\") pod \"network-metrics-daemon-s98xl\" (UID: \"d45d9164-2ba4-400d-8b27-273d7f4f351d\") " pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:07:46.995878 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.995871 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-sys\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:46.996025 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.995895 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-lib-modules\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:46.996025 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.995909 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-etc-openvswitch\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:46.996025 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.995940 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-sysctl-d\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:46.996025 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.995981 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-cni-bin\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:46.996025 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.995997 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ab0662d-ef78-415d-98f6-8b271e23e16f-konnectivity-ca\") pod \"konnectivity-agent-mz5qm\" (UID: \"3ab0662d-ef78-415d-98f6-8b271e23e16f\") " pod="kube-system/konnectivity-agent-mz5qm" Nov 02 21:07:46.996025 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996019 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-run-k8s-cni-cncf-io\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:46.996198 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996037 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-node-log\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:46.996198 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996063 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bff8d097-26a3-4621-ac01-c3b00ed23e06-ovn-node-metrics-cert\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:46.996198 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996078 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6wd7\" (UniqueName: \"kubernetes.io/projected/bff8d097-26a3-4621-ac01-c3b00ed23e06-kube-api-access-w6wd7\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:46.996198 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996103 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs\") pod \"network-metrics-daemon-s98xl\" (UID: \"d45d9164-2ba4-400d-8b27-273d7f4f351d\") " pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:07:46.996198 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996117 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-multus-conf-dir\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:46.996198 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996148 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-etc-kubernetes\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:46.996198 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996172 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-log-socket\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:46.996198 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996188 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:46.996433 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996219 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-multus-socket-dir-parent\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:46.996433 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996241 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-var-lib-kubelet\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:46.996433 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996257 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be519c9c-2677-4c4a-99c5-c537317bda3e-tmp-dir\") pod \"node-resolver-gxdgr\" (UID: \"be519c9c-2677-4c4a-99c5-c537317bda3e\") " pod="openshift-dns/node-resolver-gxdgr" Nov 02 21:07:46.996433 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996276 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:07:46.996433 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996299 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d49ad8b8-08aa-4da0-9139-b2be1efeb6cd-serviceca\") pod \"node-ca-77xn9\" (UID: \"d49ad8b8-08aa-4da0-9139-b2be1efeb6cd\") " pod="openshift-image-registry/node-ca-77xn9" Nov 02 21:07:46.996433 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996313 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-registration-dir\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:07:46.996433 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996332 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-device-dir\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:07:46.996433 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996372 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-system-cni-dir\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:46.996433 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996393 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-cnibin\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:46.996433 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996411 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-run-netns\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:46.996433 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996429 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-cnibin\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:07:46.996876 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996450 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-kubelet\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:46.996876 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996465 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-systemd-units\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:46.996876 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996515 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-run-ovn-kubernetes\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:46.996876 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996535 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bff8d097-26a3-4621-ac01-c3b00ed23e06-ovnkube-config\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:46.996876 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996548 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rchb2\" (UniqueName: \"kubernetes.io/projected/d992040e-0d1e-4227-befb-cfef1acd8aad-kube-api-access-rchb2\") pod \"iptables-alerter-zpkht\" (UID: \"d992040e-0d1e-4227-befb-cfef1acd8aad\") " pod="openshift-network-operator/iptables-alerter-zpkht" Nov 02 21:07:46.996876 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996570 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-run\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:46.996876 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996605 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx42g\" (UniqueName: \"kubernetes.io/projected/a7204d98-9f92-46e6-9e6d-482371e0a3ee-kube-api-access-sx42g\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:46.996876 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996633 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-multus-cni-dir\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:46.996876 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996668 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d49ad8b8-08aa-4da0-9139-b2be1efeb6cd-host\") pod \"node-ca-77xn9\" (UID: \"d49ad8b8-08aa-4da0-9139-b2be1efeb6cd\") " pod="openshift-image-registry/node-ca-77xn9" Nov 02 21:07:46.996876 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996708 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bff8d097-26a3-4621-ac01-c3b00ed23e06-env-overrides\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:46.996876 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996732 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-kubelet-config\") pod \"global-pull-secret-syncer-bllxh\" (UID: \"87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1\") " pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:07:46.996876 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996770 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-socket-dir\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:07:46.996876 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996795 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be519c9c-2677-4c4a-99c5-c537317bda3e-hosts-file\") pod \"node-resolver-gxdgr\" (UID: \"be519c9c-2677-4c4a-99c5-c537317bda3e\") " pod="openshift-dns/node-resolver-gxdgr" Nov 02 21:07:46.996876 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996810 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-slash\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:46.996876 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996825 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-run-ovn\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:46.996876 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996842 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-dbus\") pod \"global-pull-secret-syncer-bllxh\" (UID: \"87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1\") " pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:07:46.997330 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996889 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ab0662d-ef78-415d-98f6-8b271e23e16f-agent-certs\") pod \"konnectivity-agent-mz5qm\" (UID: \"3ab0662d-ef78-415d-98f6-8b271e23e16f\") " pod="kube-system/konnectivity-agent-mz5qm" Nov 02 21:07:46.997330 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996912 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-sys-fs\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:07:46.997330 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996936 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-systemd\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:46.997330 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996960 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fshk5\" (UniqueName: \"kubernetes.io/projected/be519c9c-2677-4c4a-99c5-c537317bda3e-kube-api-access-fshk5\") pod \"node-resolver-gxdgr\" (UID: \"be519c9c-2677-4c4a-99c5-c537317bda3e\") " pod="openshift-dns/node-resolver-gxdgr" Nov 02 21:07:46.997330 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996978 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-system-cni-dir\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:07:46.997330 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.996995 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-original-pull-secret\") pod \"global-pull-secret-syncer-bllxh\" (UID: \"87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1\") " pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:07:46.997330 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.997014 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfbgk\" (UniqueName: \"kubernetes.io/projected/c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7-kube-api-access-lfbgk\") pod \"network-check-target-8c67p\" (UID: \"c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7\") " pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:07:46.997330 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.997028 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-var-lib-cni-multus\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:46.997330 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.997065 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-hostroot\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:46.997330 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.997083 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtj26\" (UniqueName: \"kubernetes.io/projected/8ca0bae8-7539-4b19-912c-4379064ecfa2-kube-api-access-gtj26\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:46.997330 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.997096 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:07:46.997330 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.997116 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgqft\" (UniqueName: \"kubernetes.io/projected/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-kube-api-access-wgqft\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:07:46.997330 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.997166 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-run-systemd\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:46.997330 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.997202 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-modprobe-d\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:46.997330 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.997233 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-sysconfig\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:46.997330 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.997256 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7204d98-9f92-46e6-9e6d-482371e0a3ee-tmp\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:46.997810 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.997270 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-cni-netd\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:46.997810 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.997294 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d992040e-0d1e-4227-befb-cfef1acd8aad-host-slash\") pod \"iptables-alerter-zpkht\" (UID: \"d992040e-0d1e-4227-befb-cfef1acd8aad\") " pod="openshift-network-operator/iptables-alerter-zpkht" Nov 02 21:07:46.997810 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.997324 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ca0bae8-7539-4b19-912c-4379064ecfa2-cni-binary-copy\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:46.997810 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.997353 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8ca0bae8-7539-4b19-912c-4379064ecfa2-multus-daemon-config\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:46.997810 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.997373 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqv42\" (UniqueName: \"kubernetes.io/projected/d49ad8b8-08aa-4da0-9139-b2be1efeb6cd-kube-api-access-gqv42\") pod \"node-ca-77xn9\" (UID: \"d49ad8b8-08aa-4da0-9139-b2be1efeb6cd\") " pod="openshift-image-registry/node-ca-77xn9" Nov 02 21:07:46.997810 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.997389 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-run-netns\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:46.997810 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.997411 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-var-lib-kubelet\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:46.997810 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.997442 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:07:46.997810 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.997483 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-var-lib-openvswitch\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:46.997810 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:46.997536 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-etc-selinux\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:07:47.009364 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.009347 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 21:07:47.009679 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.009665 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 21:07:47.010061 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.010048 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-v96df\"" Nov 02 21:07:47.010109 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.010051 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 21:07:47.011355 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.011338 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 21:07:47.011608 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.011597 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 21:07:47.012056 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.012039 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 21:07:47.012056 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.012052 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 21:07:47.012150 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.012073 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 21:07:47.012560 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.012548 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9rn4m\"" Nov 02 21:07:47.012560 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.012557 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 21:07:47.012628 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.012548 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 21:07:47.012628 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.012549 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 21:07:47.019785 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.019770 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9vsgv\"" Nov 02 21:07:47.019785 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.019771 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-z9mmm\"" Nov 02 21:07:47.019891 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.019774 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rbqzp\"" Nov 02 21:07:47.019891 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.019797 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 21:07:47.019891 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.019826 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 21:07:47.019891 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.019842 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 21:07:47.020485 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.020471 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 21:07:47.020540 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.020514 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-t5sjc" Nov 02 21:07:47.098630 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.098605 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-var-lib-kubelet\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.098630 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.098630 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be519c9c-2677-4c4a-99c5-c537317bda3e-tmp-dir\") pod \"node-resolver-gxdgr\" (UID: \"be519c9c-2677-4c4a-99c5-c537317bda3e\") " pod="openshift-dns/node-resolver-gxdgr" Nov 02 21:07:47.098831 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.098646 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:07:47.098831 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.098661 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d49ad8b8-08aa-4da0-9139-b2be1efeb6cd-serviceca\") pod \"node-ca-77xn9\" (UID: \"d49ad8b8-08aa-4da0-9139-b2be1efeb6cd\") " pod="openshift-image-registry/node-ca-77xn9" Nov 02 21:07:47.098831 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.098674 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-registration-dir\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:07:47.098831 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.098686 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-device-dir\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:07:47.098831 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.098699 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-system-cni-dir\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.098831 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.098712 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-cnibin\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.098831 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.098718 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-var-lib-kubelet\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.098831 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.098731 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-run-netns\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.098831 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.098763 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-device-dir\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:07:47.098831 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.098772 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-run-netns\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.098831 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.098787 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-cnibin\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.098831 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.098805 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-registration-dir\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:07:47.098831 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.098827 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-cnibin\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:07:47.098831 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.098833 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-system-cni-dir\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.099429 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.098860 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-kubelet\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.099429 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.098880 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-systemd-units\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.099429 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.098862 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-cnibin\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:07:47.099429 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.098835 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:07:47.099429 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.098904 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-run-ovn-kubernetes\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.099429 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.098911 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-kubelet\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.099429 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.098928 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bff8d097-26a3-4621-ac01-c3b00ed23e06-ovnkube-config\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.099429 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.098930 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-systemd-units\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.099429 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.098951 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rchb2\" (UniqueName: \"kubernetes.io/projected/d992040e-0d1e-4227-befb-cfef1acd8aad-kube-api-access-rchb2\") pod \"iptables-alerter-zpkht\" (UID: \"d992040e-0d1e-4227-befb-cfef1acd8aad\") " pod="openshift-network-operator/iptables-alerter-zpkht" Nov 02 21:07:47.099429 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.098955 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-run-ovn-kubernetes\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.099429 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.098988 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-run\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.099429 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.098992 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be519c9c-2677-4c4a-99c5-c537317bda3e-tmp-dir\") pod \"node-resolver-gxdgr\" (UID: \"be519c9c-2677-4c4a-99c5-c537317bda3e\") " pod="openshift-dns/node-resolver-gxdgr" Nov 02 21:07:47.099429 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099006 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx42g\" (UniqueName: \"kubernetes.io/projected/a7204d98-9f92-46e6-9e6d-482371e0a3ee-kube-api-access-sx42g\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.099429 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099024 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-multus-cni-dir\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.099429 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099042 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d49ad8b8-08aa-4da0-9139-b2be1efeb6cd-host\") pod \"node-ca-77xn9\" (UID: \"d49ad8b8-08aa-4da0-9139-b2be1efeb6cd\") " pod="openshift-image-registry/node-ca-77xn9" Nov 02 21:07:47.099429 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099064 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bff8d097-26a3-4621-ac01-c3b00ed23e06-env-overrides\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.099429 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099087 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-kubelet-config\") pod \"global-pull-secret-syncer-bllxh\" (UID: \"87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1\") " pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:07:47.099976 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099103 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-multus-cni-dir\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.099976 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099108 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-socket-dir\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:07:47.099976 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099151 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be519c9c-2677-4c4a-99c5-c537317bda3e-hosts-file\") pod \"node-resolver-gxdgr\" (UID: \"be519c9c-2677-4c4a-99c5-c537317bda3e\") " pod="openshift-dns/node-resolver-gxdgr" Nov 02 21:07:47.099976 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099169 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d49ad8b8-08aa-4da0-9139-b2be1efeb6cd-serviceca\") pod \"node-ca-77xn9\" (UID: \"d49ad8b8-08aa-4da0-9139-b2be1efeb6cd\") " pod="openshift-image-registry/node-ca-77xn9" Nov 02 21:07:47.099976 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099185 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-kubelet-config\") pod \"global-pull-secret-syncer-bllxh\" (UID: \"87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1\") " pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:07:47.099976 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099223 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d49ad8b8-08aa-4da0-9139-b2be1efeb6cd-host\") pod \"node-ca-77xn9\" (UID: \"d49ad8b8-08aa-4da0-9139-b2be1efeb6cd\") " pod="openshift-image-registry/node-ca-77xn9" Nov 02 21:07:47.099976 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099230 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-run\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.099976 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099172 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-slash\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.099976 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099263 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-socket-dir\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:07:47.099976 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099268 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-run-ovn\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.099976 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099273 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be519c9c-2677-4c4a-99c5-c537317bda3e-hosts-file\") pod \"node-resolver-gxdgr\" (UID: \"be519c9c-2677-4c4a-99c5-c537317bda3e\") " pod="openshift-dns/node-resolver-gxdgr" Nov 02 21:07:47.099976 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099282 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-slash\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.099976 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099296 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-dbus\") pod \"global-pull-secret-syncer-bllxh\" (UID: \"87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1\") " pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:07:47.099976 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099319 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ab0662d-ef78-415d-98f6-8b271e23e16f-agent-certs\") pod \"konnectivity-agent-mz5qm\" (UID: \"3ab0662d-ef78-415d-98f6-8b271e23e16f\") " pod="kube-system/konnectivity-agent-mz5qm" Nov 02 21:07:47.099976 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099335 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-dbus\") pod \"global-pull-secret-syncer-bllxh\" (UID: \"87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1\") " pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:07:47.099976 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099341 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-sys-fs\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:07:47.099976 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099298 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-run-ovn\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.099976 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099366 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-systemd\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.100756 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099387 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fshk5\" (UniqueName: \"kubernetes.io/projected/be519c9c-2677-4c4a-99c5-c537317bda3e-kube-api-access-fshk5\") pod \"node-resolver-gxdgr\" (UID: \"be519c9c-2677-4c4a-99c5-c537317bda3e\") " pod="openshift-dns/node-resolver-gxdgr" Nov 02 21:07:47.100756 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099392 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-sys-fs\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:07:47.100756 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099419 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-system-cni-dir\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:07:47.100756 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099438 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-systemd\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.100756 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099443 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-original-pull-secret\") pod \"global-pull-secret-syncer-bllxh\" (UID: \"87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1\") " pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:07:47.100756 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099449 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bff8d097-26a3-4621-ac01-c3b00ed23e06-env-overrides\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.100756 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099453 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bff8d097-26a3-4621-ac01-c3b00ed23e06-ovnkube-config\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.100756 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099468 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfbgk\" (UniqueName: \"kubernetes.io/projected/c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7-kube-api-access-lfbgk\") pod \"network-check-target-8c67p\" (UID: \"c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7\") " pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:07:47.100756 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099492 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-system-cni-dir\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:07:47.100756 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099499 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-var-lib-cni-multus\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.100756 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:47.099520 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:07:47.100756 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099549 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-hostroot\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.100756 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099521 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-hostroot\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.100756 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099587 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-var-lib-cni-multus\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.100756 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:47.099592 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-original-pull-secret podName:87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1 nodeName:}" failed. No retries permitted until 2026-11-02 21:07:47.599557383 +0000 UTC m=+2.256632264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-original-pull-secret") pod "global-pull-secret-syncer-bllxh" (UID: "87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:07:47.100756 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099609 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 21:07:47.100756 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099633 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtj26\" (UniqueName: \"kubernetes.io/projected/8ca0bae8-7539-4b19-912c-4379064ecfa2-kube-api-access-gtj26\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.101497 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099659 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:07:47.101497 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099684 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgqft\" (UniqueName: \"kubernetes.io/projected/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-kube-api-access-wgqft\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:07:47.101497 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099700 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-run-systemd\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.101497 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099725 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-modprobe-d\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.101497 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099737 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-run-systemd\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.101497 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099746 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-sysconfig\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.101497 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099769 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7204d98-9f92-46e6-9e6d-482371e0a3ee-tmp\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.101497 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099783 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-cni-netd\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.101497 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099795 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d992040e-0d1e-4227-befb-cfef1acd8aad-host-slash\") pod \"iptables-alerter-zpkht\" (UID: \"d992040e-0d1e-4227-befb-cfef1acd8aad\") " pod="openshift-network-operator/iptables-alerter-zpkht" Nov 02 21:07:47.101497 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099818 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ca0bae8-7539-4b19-912c-4379064ecfa2-cni-binary-copy\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.101497 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099843 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-modprobe-d\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.101497 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099850 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-cni-netd\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.101497 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099847 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8ca0bae8-7539-4b19-912c-4379064ecfa2-multus-daemon-config\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.101497 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099882 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d992040e-0d1e-4227-befb-cfef1acd8aad-host-slash\") pod \"iptables-alerter-zpkht\" (UID: \"d992040e-0d1e-4227-befb-cfef1acd8aad\") " pod="openshift-network-operator/iptables-alerter-zpkht" Nov 02 21:07:47.101497 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099885 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqv42\" (UniqueName: \"kubernetes.io/projected/d49ad8b8-08aa-4da0-9139-b2be1efeb6cd-kube-api-access-gqv42\") pod \"node-ca-77xn9\" (UID: \"d49ad8b8-08aa-4da0-9139-b2be1efeb6cd\") " pod="openshift-image-registry/node-ca-77xn9" Nov 02 21:07:47.101497 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099914 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-run-netns\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.101497 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099939 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-var-lib-kubelet\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.102053 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099945 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-sysconfig\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.102053 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099962 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:07:47.102053 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099985 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-var-lib-openvswitch\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.102053 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.099991 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-run-netns\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.102053 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100006 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-etc-selinux\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:07:47.102053 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100027 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-host\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.102053 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100066 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-var-lib-openvswitch\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.102053 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100072 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-var-lib-kubelet\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.102053 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100110 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-etc-selinux\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:07:47.102053 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100111 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:07:47.102053 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100168 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq5t9\" (UniqueName: \"kubernetes.io/projected/42b83ba5-5e98-459f-ad4f-6a94c90212be-kube-api-access-lq5t9\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:07:47.102053 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100194 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-kubernetes\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.102053 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100216 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-var-lib-cni-bin\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.102053 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100240 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-run-multus-certs\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.102053 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100262 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bff8d097-26a3-4621-ac01-c3b00ed23e06-ovnkube-script-lib\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.102053 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100285 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:07:47.102053 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100307 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-sysctl-conf\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.102516 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100332 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-tuned\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.102516 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100355 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:07:47.102516 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100379 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-run-openvswitch\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.102516 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100394 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-run-multus-certs\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.102516 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100402 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d992040e-0d1e-4227-befb-cfef1acd8aad-iptables-alerter-script\") pod \"iptables-alerter-zpkht\" (UID: \"d992040e-0d1e-4227-befb-cfef1acd8aad\") " pod="openshift-network-operator/iptables-alerter-zpkht" Nov 02 21:07:47.102516 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100406 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ca0bae8-7539-4b19-912c-4379064ecfa2-cni-binary-copy\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.102516 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100172 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-host\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.102516 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100426 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txdkn\" (UniqueName: \"kubernetes.io/projected/d45d9164-2ba4-400d-8b27-273d7f4f351d-kube-api-access-txdkn\") pod \"network-metrics-daemon-s98xl\" (UID: \"d45d9164-2ba4-400d-8b27-273d7f4f351d\") " pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:07:47.102516 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100446 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-sys\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.102516 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100455 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-kubernetes\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.102516 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100468 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-lib-modules\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.102516 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100482 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:07:47.102516 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100489 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-etc-openvswitch\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.102516 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100496 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-var-lib-cni-bin\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.102516 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100509 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-sysctl-d\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.102516 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100531 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-cni-bin\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.102516 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100538 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-run-openvswitch\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.102936 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100554 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ab0662d-ef78-415d-98f6-8b271e23e16f-konnectivity-ca\") pod \"konnectivity-agent-mz5qm\" (UID: \"3ab0662d-ef78-415d-98f6-8b271e23e16f\") " pod="kube-system/konnectivity-agent-mz5qm" Nov 02 21:07:47.102936 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100576 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-run-k8s-cni-cncf-io\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.102936 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100598 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-node-log\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.102936 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100621 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bff8d097-26a3-4621-ac01-c3b00ed23e06-ovn-node-metrics-cert\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.102936 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100647 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6wd7\" (UniqueName: \"kubernetes.io/projected/bff8d097-26a3-4621-ac01-c3b00ed23e06-kube-api-access-w6wd7\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.102936 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100684 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs\") pod \"network-metrics-daemon-s98xl\" (UID: \"d45d9164-2ba4-400d-8b27-273d7f4f351d\") " pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:07:47.102936 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100707 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-multus-conf-dir\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.102936 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100728 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-etc-kubernetes\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.102936 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100752 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-log-socket\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.102936 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100793 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.102936 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100817 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-multus-socket-dir-parent\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.102936 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100883 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d992040e-0d1e-4227-befb-cfef1acd8aad-iptables-alerter-script\") pod \"iptables-alerter-zpkht\" (UID: \"d992040e-0d1e-4227-befb-cfef1acd8aad\") " pod="openshift-network-operator/iptables-alerter-zpkht" Nov 02 21:07:47.102936 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100893 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-multus-socket-dir-parent\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.102936 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100557 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8ca0bae8-7539-4b19-912c-4379064ecfa2-multus-daemon-config\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.102936 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.100980 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-lib-modules\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.102936 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.101001 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:07:47.102936 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.101020 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-etc-openvswitch\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.103439 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.101039 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-cni-bin\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.103439 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.101185 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-sysctl-d\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.103439 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.101210 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-sysctl-conf\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.103439 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.101230 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-etc-kubernetes\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.103439 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.101239 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:07:47.103439 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.101256 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.103439 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.101298 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-log-socket\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.103439 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:47.101328 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:07:47.103439 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.101337 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-run-k8s-cni-cncf-io\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.103439 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.101371 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-node-log\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.103439 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:47.101377 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs podName:d45d9164-2ba4-400d-8b27-273d7f4f351d nodeName:}" failed. No retries permitted until 2026-11-02 21:07:47.601361895 +0000 UTC m=+2.258436776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs") pod "network-metrics-daemon-s98xl" (UID: "d45d9164-2ba4-400d-8b27-273d7f4f351d") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:07:47.103439 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.101424 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-multus-conf-dir\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.103439 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.101448 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-sys\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.103439 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.102058 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bff8d097-26a3-4621-ac01-c3b00ed23e06-ovnkube-script-lib\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.103439 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.102121 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ab0662d-ef78-415d-98f6-8b271e23e16f-konnectivity-ca\") pod \"konnectivity-agent-mz5qm\" (UID: \"3ab0662d-ef78-415d-98f6-8b271e23e16f\") " pod="kube-system/konnectivity-agent-mz5qm" Nov 02 21:07:47.103439 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.103110 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7204d98-9f92-46e6-9e6d-482371e0a3ee-tmp\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.103439 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.103224 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bff8d097-26a3-4621-ac01-c3b00ed23e06-ovn-node-metrics-cert\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.103846 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.103264 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ab0662d-ef78-415d-98f6-8b271e23e16f-agent-certs\") pod \"konnectivity-agent-mz5qm\" (UID: \"3ab0662d-ef78-415d-98f6-8b271e23e16f\") " pod="kube-system/konnectivity-agent-mz5qm" Nov 02 21:07:47.103846 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.103359 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-tuned\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.152483 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:47.152442 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:07:47.152483 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:47.152467 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:07:47.152483 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:47.152480 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lfbgk for pod openshift-network-diagnostics/network-check-target-8c67p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:07:47.152734 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:47.152543 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7-kube-api-access-lfbgk podName:c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7 nodeName:}" failed. No retries permitted until 2026-11-02 21:07:47.652525113 +0000 UTC m=+2.309599996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lfbgk" (UniqueName: "kubernetes.io/projected/c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7-kube-api-access-lfbgk") pod "network-check-target-8c67p" (UID: "c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:07:47.154380 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.154352 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rchb2\" (UniqueName: \"kubernetes.io/projected/d992040e-0d1e-4227-befb-cfef1acd8aad-kube-api-access-rchb2\") pod \"iptables-alerter-zpkht\" (UID: \"d992040e-0d1e-4227-befb-cfef1acd8aad\") " pod="openshift-network-operator/iptables-alerter-zpkht" Nov 02 21:07:47.154497 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.154378 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx42g\" (UniqueName: \"kubernetes.io/projected/a7204d98-9f92-46e6-9e6d-482371e0a3ee-kube-api-access-sx42g\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.154797 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.154772 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtj26\" (UniqueName: \"kubernetes.io/projected/8ca0bae8-7539-4b19-912c-4379064ecfa2-kube-api-access-gtj26\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.154896 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.154779 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqv42\" (UniqueName: \"kubernetes.io/projected/d49ad8b8-08aa-4da0-9139-b2be1efeb6cd-kube-api-access-gqv42\") pod \"node-ca-77xn9\" (UID: \"d49ad8b8-08aa-4da0-9139-b2be1efeb6cd\") " pod="openshift-image-registry/node-ca-77xn9" Nov 02 21:07:47.155396 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.155378 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txdkn\" (UniqueName: \"kubernetes.io/projected/d45d9164-2ba4-400d-8b27-273d7f4f351d-kube-api-access-txdkn\") pod \"network-metrics-daemon-s98xl\" (UID: \"d45d9164-2ba4-400d-8b27-273d7f4f351d\") " pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:07:47.155487 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.155467 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6wd7\" (UniqueName: \"kubernetes.io/projected/bff8d097-26a3-4621-ac01-c3b00ed23e06-kube-api-access-w6wd7\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.155657 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.155643 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq5t9\" (UniqueName: \"kubernetes.io/projected/42b83ba5-5e98-459f-ad4f-6a94c90212be-kube-api-access-lq5t9\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:07:47.156247 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.156229 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgqft\" (UniqueName: \"kubernetes.io/projected/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-kube-api-access-wgqft\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:07:47.156460 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.156446 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fshk5\" (UniqueName: \"kubernetes.io/projected/be519c9c-2677-4c4a-99c5-c537317bda3e-kube-api-access-fshk5\") pod \"node-resolver-gxdgr\" (UID: \"be519c9c-2677-4c4a-99c5-c537317bda3e\") " pod="openshift-dns/node-resolver-gxdgr" Nov 02 21:07:47.256980 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.256896 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:07:47.272414 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.272385 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zpkht" Nov 02 21:07:47.292097 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.292075 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:07:47.305492 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.305466 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-77xn9" Nov 02 21:07:47.307633 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.307616 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6blt8" Nov 02 21:07:47.324894 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.324830 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gxdgr" Nov 02 21:07:47.359395 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.359375 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:07:47.362450 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.362430 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mz5qm" Nov 02 21:07:47.364008 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:47.363985 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30622891ad10d753b5b3452735f59810.slice/crio-0ebda27314aede86920f44c514e1aeb2348dcaf88ab8704a9d68111a733f1b16 WatchSource:0}: Error finding container 0ebda27314aede86920f44c514e1aeb2348dcaf88ab8704a9d68111a733f1b16: Status 404 returned error can't find the container with id 0ebda27314aede86920f44c514e1aeb2348dcaf88ab8704a9d68111a733f1b16 Nov 02 21:07:47.365219 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:47.365199 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398d07e61029ddf8dea61a5829612cad.slice/crio-ed611eb381e81907dcc1e8778781305f8612bdde748f9c61c70645463827e131 WatchSource:0}: Error finding container ed611eb381e81907dcc1e8778781305f8612bdde748f9c61c70645463827e131: Status 404 returned error can't find the container with id ed611eb381e81907dcc1e8778781305f8612bdde748f9c61c70645463827e131 Nov 02 21:07:47.369239 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.369082 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:07:47.369601 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.369579 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:07:47.371225 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:47.371208 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff8d097_26a3_4621_ac01_c3b00ed23e06.slice/crio-dc462923466524e209c628f11d6b36ff1698809fd6c547d6b1821ccb8fb05c0d WatchSource:0}: Error finding container dc462923466524e209c628f11d6b36ff1698809fd6c547d6b1821ccb8fb05c0d: Status 404 returned error can't find the container with id dc462923466524e209c628f11d6b36ff1698809fd6c547d6b1821ccb8fb05c0d Nov 02 21:07:47.372004 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:47.371971 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab0662d_ef78_415d_98f6_8b271e23e16f.slice/crio-67621105d0befe73aa74ba7e09f62ac325157669fa22233822aafe5b1fb981f2 WatchSource:0}: Error finding container 67621105d0befe73aa74ba7e09f62ac325157669fa22233822aafe5b1fb981f2: Status 404 returned error can't find the container with id 67621105d0befe73aa74ba7e09f62ac325157669fa22233822aafe5b1fb981f2 Nov 02 21:07:47.378349 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:47.378331 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b83ba5_5e98_459f_ad4f_6a94c90212be.slice/crio-d189fe9831262764a041fc7b52264ccceae18308ea1905cab9e7ce1a790e5ed7 WatchSource:0}: Error finding container d189fe9831262764a041fc7b52264ccceae18308ea1905cab9e7ce1a790e5ed7: Status 404 returned error can't find the container with id d189fe9831262764a041fc7b52264ccceae18308ea1905cab9e7ce1a790e5ed7 Nov 02 21:07:47.605508 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.605479 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-original-pull-secret\") pod \"global-pull-secret-syncer-bllxh\" (UID: \"87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1\") " pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:07:47.605717 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.605530 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs\") pod \"network-metrics-daemon-s98xl\" (UID: \"d45d9164-2ba4-400d-8b27-273d7f4f351d\") " pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:07:47.605717 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:47.605612 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:07:47.605717 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:47.605657 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs podName:d45d9164-2ba4-400d-8b27-273d7f4f351d nodeName:}" failed. No retries permitted until 2026-11-02 21:07:48.605644145 +0000 UTC m=+3.262719011 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs") pod "network-metrics-daemon-s98xl" (UID: "d45d9164-2ba4-400d-8b27-273d7f4f351d") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:07:47.605717 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:47.605659 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:07:47.605717 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:47.605717 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-original-pull-secret podName:87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1 nodeName:}" failed. No retries permitted until 2026-11-02 21:07:48.605698467 +0000 UTC m=+3.262773349 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-original-pull-secret") pod "global-pull-secret-syncer-bllxh" (UID: "87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:07:47.706539 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.706508 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfbgk\" (UniqueName: \"kubernetes.io/projected/c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7-kube-api-access-lfbgk\") pod \"network-check-target-8c67p\" (UID: \"c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7\") " pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:07:47.707028 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:47.706670 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:07:47.707028 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:47.706690 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:07:47.707028 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:47.706699 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lfbgk for pod openshift-network-diagnostics/network-check-target-8c67p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:07:47.707028 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:47.706748 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7-kube-api-access-lfbgk podName:c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7 nodeName:}" failed. No retries permitted until 2026-11-02 21:07:48.706733229 +0000 UTC m=+3.363808095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfbgk" (UniqueName: "kubernetes.io/projected/c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7-kube-api-access-lfbgk") pod "network-check-target-8c67p" (UID: "c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:07:47.769933 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.769878 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" event={"ID":"42b83ba5-5e98-459f-ad4f-6a94c90212be","Type":"ContainerStarted","Data":"d189fe9831262764a041fc7b52264ccceae18308ea1905cab9e7ce1a790e5ed7"} Nov 02 21:07:47.770839 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.770814 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mz5qm" event={"ID":"3ab0662d-ef78-415d-98f6-8b271e23e16f","Type":"ContainerStarted","Data":"67621105d0befe73aa74ba7e09f62ac325157669fa22233822aafe5b1fb981f2"} Nov 02 21:07:47.771811 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.771792 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" event={"ID":"bff8d097-26a3-4621-ac01-c3b00ed23e06","Type":"ContainerStarted","Data":"dc462923466524e209c628f11d6b36ff1698809fd6c547d6b1821ccb8fb05c0d"} Nov 02 21:07:47.772685 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.772660 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-89.us-west-2.compute.internal" event={"ID":"398d07e61029ddf8dea61a5829612cad","Type":"ContainerStarted","Data":"ed611eb381e81907dcc1e8778781305f8612bdde748f9c61c70645463827e131"} Nov 02 21:07:47.773565 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.773547 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal" event={"ID":"30622891ad10d753b5b3452735f59810","Type":"ContainerStarted","Data":"0ebda27314aede86920f44c514e1aeb2348dcaf88ab8704a9d68111a733f1b16"} Nov 02 21:07:47.860504 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.860431 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:07:47.942610 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:47.942582 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:07:48.022946 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:48.022881 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:02:46 +0000 UTC" deadline="2028-04-29 03:50:54.703532381 +0000 UTC" Nov 02 21:07:48.022946 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:48.022935 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13038h43m6.680599723s" Nov 02 21:07:48.084062 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:48.084035 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:07:48.090552 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:48.090523 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd992040e_0d1e_4227_befb_cfef1acd8aad.slice/crio-46114be710d5a4afe8f3e6fb9d6da86b2d9882a4ad235f92ef80f3710c460d13 WatchSource:0}: Error finding container 46114be710d5a4afe8f3e6fb9d6da86b2d9882a4ad235f92ef80f3710c460d13: Status 404 returned error can't find the container with id 46114be710d5a4afe8f3e6fb9d6da86b2d9882a4ad235f92ef80f3710c460d13 Nov 02 21:07:48.091340 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:48.091309 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dab1353_88e8_4250_b7ab_a4b2a69e37fa.slice/crio-7f401984caa01a6a03fd60b1964dcc5d3ff649eccb524d2222249b131590e1b8 WatchSource:0}: Error finding container 7f401984caa01a6a03fd60b1964dcc5d3ff649eccb524d2222249b131590e1b8: Status 404 returned error can't find the container with id 7f401984caa01a6a03fd60b1964dcc5d3ff649eccb524d2222249b131590e1b8 Nov 02 21:07:48.117472 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:48.117446 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7204d98_9f92_46e6_9e6d_482371e0a3ee.slice/crio-13a7f9180c1da924b1071e6e29d31cd306309d4c064227e59cff2cf5ccf6e20e WatchSource:0}: Error finding container 13a7f9180c1da924b1071e6e29d31cd306309d4c064227e59cff2cf5ccf6e20e: Status 404 returned error can't find the container with id 13a7f9180c1da924b1071e6e29d31cd306309d4c064227e59cff2cf5ccf6e20e Nov 02 21:07:48.167032 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:48.166984 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca0bae8_7539_4b19_912c_4379064ecfa2.slice/crio-01c2aa740e152759c0ae026d7d14853bd1ce15949ee48d84b6c1118358ec89dd WatchSource:0}: Error finding container 01c2aa740e152759c0ae026d7d14853bd1ce15949ee48d84b6c1118358ec89dd: Status 404 returned error can't find the container with id 01c2aa740e152759c0ae026d7d14853bd1ce15949ee48d84b6c1118358ec89dd Nov 02 21:07:48.186328 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:48.186301 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49ad8b8_08aa_4da0_9139_b2be1efeb6cd.slice/crio-77adee5b179ff360088237e14769bbde2edd21eab5ed31dca582284317167945 WatchSource:0}: Error finding container 77adee5b179ff360088237e14769bbde2edd21eab5ed31dca582284317167945: Status 404 returned error can't find the container with id 77adee5b179ff360088237e14769bbde2edd21eab5ed31dca582284317167945 Nov 02 21:07:48.231969 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:07:48.231943 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe519c9c_2677_4c4a_99c5_c537317bda3e.slice/crio-b7afbd2ef8c2fd552a7bde8d073aae109ba33a010abf7b03f5689b7c6f7c8589 WatchSource:0}: Error finding container b7afbd2ef8c2fd552a7bde8d073aae109ba33a010abf7b03f5689b7c6f7c8589: Status 404 returned error can't find the container with id b7afbd2ef8c2fd552a7bde8d073aae109ba33a010abf7b03f5689b7c6f7c8589 Nov 02 21:07:48.612154 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:48.611681 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs\") pod \"network-metrics-daemon-s98xl\" (UID: \"d45d9164-2ba4-400d-8b27-273d7f4f351d\") " pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:07:48.612154 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:48.611748 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-original-pull-secret\") pod \"global-pull-secret-syncer-bllxh\" (UID: \"87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1\") " pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:07:48.612154 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:48.611874 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:07:48.612154 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:48.611936 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-original-pull-secret podName:87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1 nodeName:}" failed. No retries permitted until 2026-11-02 21:07:50.611917044 +0000 UTC m=+5.268991917 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-original-pull-secret") pod "global-pull-secret-syncer-bllxh" (UID: "87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:07:48.612154 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:48.612003 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:07:48.612154 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:48.612036 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs podName:d45d9164-2ba4-400d-8b27-273d7f4f351d nodeName:}" failed. No retries permitted until 2026-11-02 21:07:50.612024444 +0000 UTC m=+5.269099319 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs") pod "network-metrics-daemon-s98xl" (UID: "d45d9164-2ba4-400d-8b27-273d7f4f351d") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:07:48.712891 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:48.712555 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfbgk\" (UniqueName: \"kubernetes.io/projected/c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7-kube-api-access-lfbgk\") pod \"network-check-target-8c67p\" (UID: \"c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7\") " pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:07:48.712891 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:48.712765 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:07:48.712891 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:48.712783 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:07:48.712891 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:48.712796 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lfbgk for pod openshift-network-diagnostics/network-check-target-8c67p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:07:48.712891 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:48.712855 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7-kube-api-access-lfbgk podName:c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7 nodeName:}" failed. No retries permitted until 2026-11-02 21:07:50.712836755 +0000 UTC m=+5.369911629 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfbgk" (UniqueName: "kubernetes.io/projected/c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7-kube-api-access-lfbgk") pod "network-check-target-8c67p" (UID: "c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:07:48.766771 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:48.766737 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:07:48.766919 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:48.766869 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c67p" podUID="c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7" Nov 02 21:07:48.766991 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:48.766965 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:07:48.767258 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:48.767048 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bllxh" podUID="87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1" Nov 02 21:07:48.767258 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:48.767117 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:07:48.767258 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:48.767213 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s98xl" podUID="d45d9164-2ba4-400d-8b27-273d7f4f351d" Nov 02 21:07:48.789471 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:48.789375 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gxdgr" event={"ID":"be519c9c-2677-4c4a-99c5-c537317bda3e","Type":"ContainerStarted","Data":"b7afbd2ef8c2fd552a7bde8d073aae109ba33a010abf7b03f5689b7c6f7c8589"} Nov 02 21:07:48.794415 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:48.794338 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" event={"ID":"a7204d98-9f92-46e6-9e6d-482371e0a3ee","Type":"ContainerStarted","Data":"13a7f9180c1da924b1071e6e29d31cd306309d4c064227e59cff2cf5ccf6e20e"} Nov 02 21:07:48.803068 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:48.802993 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94zch" event={"ID":"7dab1353-88e8-4250-b7ab-a4b2a69e37fa","Type":"ContainerStarted","Data":"7f401984caa01a6a03fd60b1964dcc5d3ff649eccb524d2222249b131590e1b8"} Nov 02 21:07:48.806346 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:48.806314 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zpkht" event={"ID":"d992040e-0d1e-4227-befb-cfef1acd8aad","Type":"ContainerStarted","Data":"46114be710d5a4afe8f3e6fb9d6da86b2d9882a4ad235f92ef80f3710c460d13"} Nov 02 21:07:48.813190 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:48.813146 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-77xn9" event={"ID":"d49ad8b8-08aa-4da0-9139-b2be1efeb6cd","Type":"ContainerStarted","Data":"77adee5b179ff360088237e14769bbde2edd21eab5ed31dca582284317167945"} Nov 02 21:07:48.818021 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:48.817994 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6blt8" event={"ID":"8ca0bae8-7539-4b19-912c-4379064ecfa2","Type":"ContainerStarted","Data":"01c2aa740e152759c0ae026d7d14853bd1ce15949ee48d84b6c1118358ec89dd"} Nov 02 21:07:49.023753 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:49.023686 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:02:46 +0000 UTC" deadline="2028-07-09 01:47:15.534261736 +0000 UTC" Nov 02 21:07:49.023753 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:49.023751 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14740h39m26.51051442s" Nov 02 21:07:50.630685 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:50.630271 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs\") pod \"network-metrics-daemon-s98xl\" (UID: \"d45d9164-2ba4-400d-8b27-273d7f4f351d\") " pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:07:50.630685 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:50.630334 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-original-pull-secret\") pod \"global-pull-secret-syncer-bllxh\" (UID: \"87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1\") " pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:07:50.630685 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:50.630481 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:07:50.630685 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:50.630537 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-original-pull-secret podName:87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1 nodeName:}" failed. No retries permitted until 2026-11-02 21:07:54.630520256 +0000 UTC m=+9.287595130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-original-pull-secret") pod "global-pull-secret-syncer-bllxh" (UID: "87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:07:50.630685 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:50.630604 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:07:50.630685 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:50.630638 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs podName:d45d9164-2ba4-400d-8b27-273d7f4f351d nodeName:}" failed. No retries permitted until 2026-11-02 21:07:54.630626151 +0000 UTC m=+9.287701025 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs") pod "network-metrics-daemon-s98xl" (UID: "d45d9164-2ba4-400d-8b27-273d7f4f351d") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:07:50.731380 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:50.731340 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfbgk\" (UniqueName: \"kubernetes.io/projected/c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7-kube-api-access-lfbgk\") pod \"network-check-target-8c67p\" (UID: \"c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7\") " pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:07:50.731562 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:50.731497 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:07:50.731562 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:50.731522 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:07:50.731562 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:50.731534 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lfbgk for pod openshift-network-diagnostics/network-check-target-8c67p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:07:50.731727 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:50.731589 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7-kube-api-access-lfbgk podName:c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7 nodeName:}" failed. No retries permitted until 2026-11-02 21:07:54.731573369 +0000 UTC m=+9.388648237 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfbgk" (UniqueName: "kubernetes.io/projected/c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7-kube-api-access-lfbgk") pod "network-check-target-8c67p" (UID: "c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:07:50.766203 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:50.766164 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:07:50.766382 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:50.766315 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c67p" podUID="c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7" Nov 02 21:07:50.766455 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:50.766382 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:07:50.766516 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:50.766494 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bllxh" podUID="87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1" Nov 02 21:07:50.766569 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:50.766543 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:07:50.766660 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:50.766623 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s98xl" podUID="d45d9164-2ba4-400d-8b27-273d7f4f351d" Nov 02 21:07:52.766488 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:52.766243 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:07:52.767059 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:52.766600 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bllxh" podUID="87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1" Nov 02 21:07:52.767059 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:52.766694 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:07:52.767059 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:52.766772 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s98xl" podUID="d45d9164-2ba4-400d-8b27-273d7f4f351d" Nov 02 21:07:52.767059 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:52.766820 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:07:52.767059 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:52.766881 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c67p" podUID="c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7" Nov 02 21:07:52.847333 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:52.847257 2577 generic.go:356] "Generic (PLEG): container finished" podID="7dab1353-88e8-4250-b7ab-a4b2a69e37fa" containerID="a3b923e4c2fa18f9821bd7e6344c8f9147e38b3f84e726b9fa40cf8345ab57ba" exitCode=0 Nov 02 21:07:52.847490 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:52.847335 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94zch" event={"ID":"7dab1353-88e8-4250-b7ab-a4b2a69e37fa","Type":"ContainerDied","Data":"a3b923e4c2fa18f9821bd7e6344c8f9147e38b3f84e726b9fa40cf8345ab57ba"} Nov 02 21:07:52.851475 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:52.851446 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mz5qm" event={"ID":"3ab0662d-ef78-415d-98f6-8b271e23e16f","Type":"ContainerStarted","Data":"2dc3e2f2c25e703561d16e66127674c6464b63d96629d0d0b20cd15ae9998412"} Nov 02 21:07:52.863448 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:52.863415 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-89.us-west-2.compute.internal" event={"ID":"398d07e61029ddf8dea61a5829612cad","Type":"ContainerStarted","Data":"532f778c076c1a2b47d6ad924e142963e3d95997f7e5a7b4d7dcafffe92ac96f"} Nov 02 21:07:52.868058 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:52.868026 2577 generic.go:356] "Generic (PLEG): container finished" podID="30622891ad10d753b5b3452735f59810" containerID="39b9b34b579ceea3f81bca21394f7c2208287cfd9eee929286d43c55f6c4818e" exitCode=0 Nov 02 21:07:52.868196 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:52.868074 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal" event={"ID":"30622891ad10d753b5b3452735f59810","Type":"ContainerDied","Data":"39b9b34b579ceea3f81bca21394f7c2208287cfd9eee929286d43c55f6c4818e"} Nov 02 21:07:53.191002 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:53.190901 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-89.us-west-2.compute.internal" podStartSLOduration=7.190883328 podStartE2EDuration="7.190883328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:07:53.190082921 +0000 UTC m=+7.847157811" watchObservedRunningTime="2026-11-02 21:07:53.190883328 +0000 UTC m=+7.847958219" Nov 02 21:07:53.305985 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:53.305931 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mz5qm" podStartSLOduration=2.754711431 podStartE2EDuration="7.305918651s" totalImagesPullingTime="4.55120722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:07:53.305311659 +0000 UTC m=+7.962386550" watchObservedRunningTime="2026-11-02 21:07:53.305918651 +0000 UTC m=+7.962993541" Nov 02 21:07:53.679074 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:53.678957 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mz5qm" Nov 02 21:07:53.679940 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:53.679755 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mz5qm" Nov 02 21:07:53.875118 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:53.874406 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal" event={"ID":"30622891ad10d753b5b3452735f59810","Type":"ContainerStarted","Data":"0e96a624cbb371d3b047de96fca96c163d686d7008f4a1284be47bb01eed6b18"} Nov 02 21:07:54.669308 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:54.669267 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs\") pod \"network-metrics-daemon-s98xl\" (UID: \"d45d9164-2ba4-400d-8b27-273d7f4f351d\") " pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:07:54.669509 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:54.669345 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-original-pull-secret\") pod \"global-pull-secret-syncer-bllxh\" (UID: \"87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1\") " pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:07:54.669509 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:54.669478 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:07:54.669631 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:54.669538 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-original-pull-secret podName:87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:02.669519968 +0000 UTC m=+17.326594850 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-original-pull-secret") pod "global-pull-secret-syncer-bllxh" (UID: "87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:07:54.669631 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:54.669603 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:07:54.669715 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:54.669635 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs podName:d45d9164-2ba4-400d-8b27-273d7f4f351d nodeName:}" failed. No retries permitted until 2026-11-02 21:08:02.669623992 +0000 UTC m=+17.326698865 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs") pod "network-metrics-daemon-s98xl" (UID: "d45d9164-2ba4-400d-8b27-273d7f4f351d") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:07:54.766969 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:54.766936 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:07:54.767175 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:54.767067 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c67p" podUID="c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7" Nov 02 21:07:54.767175 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:54.767153 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:07:54.767284 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:54.767234 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bllxh" podUID="87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1" Nov 02 21:07:54.767318 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:54.767293 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:07:54.767394 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:54.767371 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s98xl" podUID="d45d9164-2ba4-400d-8b27-273d7f4f351d" Nov 02 21:07:54.769769 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:54.769742 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfbgk\" (UniqueName: \"kubernetes.io/projected/c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7-kube-api-access-lfbgk\") pod \"network-check-target-8c67p\" (UID: \"c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7\") " pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:07:54.769951 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:54.769934 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:07:54.770026 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:54.769958 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:07:54.770026 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:54.769972 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lfbgk for pod openshift-network-diagnostics/network-check-target-8c67p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:07:54.770026 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:54.770024 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7-kube-api-access-lfbgk podName:c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:02.77000742 +0000 UTC m=+17.427082304 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfbgk" (UniqueName: "kubernetes.io/projected/c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7-kube-api-access-lfbgk") pod "network-check-target-8c67p" (UID: "c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:07:54.880814 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:54.880773 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" event={"ID":"42b83ba5-5e98-459f-ad4f-6a94c90212be","Type":"ContainerStarted","Data":"d5923710ec0120c42762bc053ff4bd10afbd634eed9769d5264d36e758f95d50"} Nov 02 21:07:54.884611 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:54.883142 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 21:07:54.884611 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:54.884027 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-77xn9" event={"ID":"d49ad8b8-08aa-4da0-9139-b2be1efeb6cd","Type":"ContainerStarted","Data":"424779b7b6818fbb2781a5da37c5c01117e2bc229463ce8ae720976fdc4682f6"} Nov 02 21:07:54.998513 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:54.998406 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal" podStartSLOduration=8.998383958 podStartE2EDuration="8.998383958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:07:53.9926551 +0000 UTC m=+8.649729988" watchObservedRunningTime="2026-11-02 21:07:54.998383958 +0000 UTC m=+9.655458848" Nov 02 21:07:56.173091 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:56.173056 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 21:07:56.766044 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:56.766008 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:07:56.766218 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:56.766164 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bllxh" podUID="87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1" Nov 02 21:07:56.766586 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:56.766571 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:07:56.766686 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:56.766669 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s98xl" podUID="d45d9164-2ba4-400d-8b27-273d7f4f351d" Nov 02 21:07:56.766733 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:56.766721 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:07:56.766791 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:56.766779 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c67p" podUID="c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7" Nov 02 21:07:56.771060 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:56.770974 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T21:07:56.173076791Z","UUID":"48c0989d-2c35-4390-b9a4-c1e9c472d235","Handler":null,"Name":"","Endpoint":""} Nov 02 21:07:56.772855 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:56.772824 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 21:07:56.772855 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:56.772856 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 21:07:56.900761 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:56.900720 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6blt8" event={"ID":"8ca0bae8-7539-4b19-912c-4379064ecfa2","Type":"ContainerStarted","Data":"8070285a6e052da58fd0ab999180dc375d77f0ac7ddce3d39ebaaa5027df95fd"} Nov 02 21:07:56.903222 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:56.903193 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gxdgr" event={"ID":"be519c9c-2677-4c4a-99c5-c537317bda3e","Type":"ContainerStarted","Data":"063342f21546ed14d61630df09a3792cea49fae86c8c895a78131cc1635a44e4"} Nov 02 21:07:56.905734 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:56.905707 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" event={"ID":"42b83ba5-5e98-459f-ad4f-6a94c90212be","Type":"ContainerStarted","Data":"4a1c0f27a277754102d3aaf141b23836814aeca17450b49a08ec343644126ab1"} Nov 02 21:07:57.018101 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:57.018001 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-77xn9" podStartSLOduration=5.175915366 podStartE2EDuration="11.017982718s" totalImagesPullingTime="5.842067352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:07:54.999353543 +0000 UTC m=+9.656428437" watchObservedRunningTime="2026-11-02 21:07:57.017982718 +0000 UTC m=+11.675057609" Nov 02 21:07:57.018261 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:57.018173 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-6blt8" podStartSLOduration=3.04680354 podStartE2EDuration="11.018163265s" totalImagesPullingTime="7.971359725s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:07:57.017518628 +0000 UTC m=+11.674593518" watchObservedRunningTime="2026-11-02 21:07:57.018163265 +0000 UTC m=+11.675238157" Nov 02 21:07:57.136618 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:57.135836 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-gxdgr" podStartSLOduration=2.901532276 podStartE2EDuration="11.135825103s" totalImagesPullingTime="8.234292827s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:07:57.135766805 +0000 UTC m=+11.792841696" watchObservedRunningTime="2026-11-02 21:07:57.135825103 +0000 UTC m=+11.792899993" Nov 02 21:07:57.909732 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:57.909699 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zpkht" event={"ID":"d992040e-0d1e-4227-befb-cfef1acd8aad","Type":"ContainerStarted","Data":"01c6deb024eef472264038b89ea970c6494c16ab1b9121ce7a1869c722aa439f"} Nov 02 21:07:57.916750 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:57.916717 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" event={"ID":"42b83ba5-5e98-459f-ad4f-6a94c90212be","Type":"ContainerStarted","Data":"f2a9307e3fb9799e7c4beb6f72458646126fceb0d75030e42c26947b968d1827"} Nov 02 21:07:58.043112 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:58.043056 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zpkht" podStartSLOduration=4.987387066 podStartE2EDuration="13.043039813s" totalImagesPullingTime="8.055652747s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:07:58.042518578 +0000 UTC m=+12.699593467" watchObservedRunningTime="2026-11-02 21:07:58.043039813 +0000 UTC m=+12.700114702" Nov 02 21:07:58.163733 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:58.163665 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" podStartSLOduration=2.179985547 podStartE2EDuration="12.163646974s" totalImagesPullingTime="9.983661427s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:07:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:07:58.162784558 +0000 UTC m=+12.819859448" watchObservedRunningTime="2026-11-02 21:07:58.163646974 +0000 UTC m=+12.820721866" Nov 02 21:07:58.766181 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:58.765955 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:07:58.766181 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:58.766189 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:07:58.766386 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:58.766159 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:07:58.766386 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:58.766306 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c67p" podUID="c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7" Nov 02 21:07:58.766386 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:58.766336 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bllxh" podUID="87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1" Nov 02 21:07:58.766510 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:07:58.766410 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s98xl" podUID="d45d9164-2ba4-400d-8b27-273d7f4f351d" Nov 02 21:07:58.924772 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:58.924683 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" event={"ID":"a7204d98-9f92-46e6-9e6d-482371e0a3ee","Type":"ContainerStarted","Data":"40be74b96b6769b743e1e0b8f9e0c26f547bc0895a25280821f9865272a1c29f"} Nov 02 21:07:58.926740 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:58.926711 2577 generic.go:356] "Generic (PLEG): container finished" podID="7dab1353-88e8-4250-b7ab-a4b2a69e37fa" containerID="cbe28e728a99a4cdaf6d48995183b5e93e447192107c156b3e83a3d177f8615e" exitCode=0 Nov 02 21:07:58.926854 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:58.926801 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94zch" event={"ID":"7dab1353-88e8-4250-b7ab-a4b2a69e37fa","Type":"ContainerDied","Data":"cbe28e728a99a4cdaf6d48995183b5e93e447192107c156b3e83a3d177f8615e"} Nov 02 21:07:59.044280 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:07:59.044220 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" podStartSLOduration=2.812805655 podStartE2EDuration="13.04418465s" totalImagesPullingTime="10.231378995s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:07:59.043850759 +0000 UTC m=+13.700925647" watchObservedRunningTime="2026-11-02 21:07:59.04418465 +0000 UTC m=+13.701259543" Nov 02 21:08:00.766310 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:00.766269 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:08:00.766838 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:00.766276 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:08:00.766838 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:00.766413 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:08:00.766838 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:00.766413 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s98xl" podUID="d45d9164-2ba4-400d-8b27-273d7f4f351d" Nov 02 21:08:00.766838 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:00.766493 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bllxh" podUID="87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1" Nov 02 21:08:00.766838 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:00.766568 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c67p" podUID="c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7" Nov 02 21:08:00.933581 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:00.933549 2577 generic.go:356] "Generic (PLEG): container finished" podID="7dab1353-88e8-4250-b7ab-a4b2a69e37fa" containerID="729d877e2b17c4de601d4f03d6d1ae62895b3d7e6990325390e987bd77f855a5" exitCode=0 Nov 02 21:08:00.933757 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:00.933590 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94zch" event={"ID":"7dab1353-88e8-4250-b7ab-a4b2a69e37fa","Type":"ContainerDied","Data":"729d877e2b17c4de601d4f03d6d1ae62895b3d7e6990325390e987bd77f855a5"} Nov 02 21:08:02.732856 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:02.732690 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs\") pod \"network-metrics-daemon-s98xl\" (UID: \"d45d9164-2ba4-400d-8b27-273d7f4f351d\") " pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:08:02.733160 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:02.732883 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-original-pull-secret\") pod \"global-pull-secret-syncer-bllxh\" (UID: \"87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1\") " pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:08:02.733160 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:02.732826 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:08:02.733160 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:02.732982 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:08:02.733160 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:02.732986 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs podName:d45d9164-2ba4-400d-8b27-273d7f4f351d nodeName:}" failed. No retries permitted until 2026-11-02 21:08:18.732968268 +0000 UTC m=+33.390043136 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs") pod "network-metrics-daemon-s98xl" (UID: "d45d9164-2ba4-400d-8b27-273d7f4f351d") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:08:02.733160 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:02.733023 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-original-pull-secret podName:87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:18.733015664 +0000 UTC m=+33.390090535 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-original-pull-secret") pod "global-pull-secret-syncer-bllxh" (UID: "87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:08:02.765856 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:02.765832 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:08:02.765856 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:02.765847 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:08:02.766024 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:02.765970 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s98xl" podUID="d45d9164-2ba4-400d-8b27-273d7f4f351d" Nov 02 21:08:02.766024 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:02.766009 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:08:02.766141 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:02.766085 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bllxh" podUID="87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1" Nov 02 21:08:02.766208 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:02.766177 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c67p" podUID="c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7" Nov 02 21:08:02.834019 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:02.833991 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfbgk\" (UniqueName: \"kubernetes.io/projected/c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7-kube-api-access-lfbgk\") pod \"network-check-target-8c67p\" (UID: \"c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7\") " pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:08:02.834144 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:02.834119 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:08:02.834182 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:02.834153 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:08:02.834182 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:02.834163 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lfbgk for pod openshift-network-diagnostics/network-check-target-8c67p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:08:02.834235 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:02.834211 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7-kube-api-access-lfbgk podName:c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:18.83419901 +0000 UTC m=+33.491273891 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfbgk" (UniqueName: "kubernetes.io/projected/c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7-kube-api-access-lfbgk") pod "network-check-target-8c67p" (UID: "c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:08:02.941098 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:02.940944 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 21:08:02.941468 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:02.941444 2577 generic.go:356] "Generic (PLEG): container finished" podID="bff8d097-26a3-4621-ac01-c3b00ed23e06" containerID="c6cb2944006d1ee69c92ed1bb854345a65049bfff5f2cf4a817992c7eb338b8e" exitCode=1 Nov 02 21:08:02.941537 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:02.941516 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" event={"ID":"bff8d097-26a3-4621-ac01-c3b00ed23e06","Type":"ContainerStarted","Data":"c1c7257e6e8517f4d946cf41f7f3eb896cf075e82094ceb5c52cb26818c37684"} Nov 02 21:08:02.941616 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:02.941603 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" event={"ID":"bff8d097-26a3-4621-ac01-c3b00ed23e06","Type":"ContainerStarted","Data":"dabae457e7a06035de907c95e6279790827c97e6ec2a372514181a9e2e79fb00"} Nov 02 21:08:02.941674 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:02.941660 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" event={"ID":"bff8d097-26a3-4621-ac01-c3b00ed23e06","Type":"ContainerStarted","Data":"d533135b1c83d5f2b71e9dee577c2eaadc6196e96c8e8add30b9b59dfb988cf8"} Nov 02 21:08:02.941724 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:02.941709 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" event={"ID":"bff8d097-26a3-4621-ac01-c3b00ed23e06","Type":"ContainerDied","Data":"c6cb2944006d1ee69c92ed1bb854345a65049bfff5f2cf4a817992c7eb338b8e"} Nov 02 21:08:02.941772 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:02.941758 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" event={"ID":"bff8d097-26a3-4621-ac01-c3b00ed23e06","Type":"ContainerStarted","Data":"fd18a84a078a0378982212400459097bef3574cb5cf2f3a98d5a5cfc9a615d01"} Nov 02 21:08:02.943902 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:02.943881 2577 generic.go:356] "Generic (PLEG): container finished" podID="7dab1353-88e8-4250-b7ab-a4b2a69e37fa" containerID="eb2a444d2a339d2a1c6a991b86c8ace00f8093a01d10dca33f604c755ccaaae7" exitCode=0 Nov 02 21:08:02.943991 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:02.943921 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94zch" event={"ID":"7dab1353-88e8-4250-b7ab-a4b2a69e37fa","Type":"ContainerDied","Data":"eb2a444d2a339d2a1c6a991b86c8ace00f8093a01d10dca33f604c755ccaaae7"} Nov 02 21:08:03.948830 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:03.948799 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 21:08:03.949278 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:03.949156 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" event={"ID":"bff8d097-26a3-4621-ac01-c3b00ed23e06","Type":"ContainerStarted","Data":"a7327728b76adbb115c2976e5cfb0c5c2abf1c209404f7dae6abffd9719dc1cb"} Nov 02 21:08:04.766079 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:04.766034 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:08:04.766267 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:04.766180 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:08:04.766320 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:04.766298 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c67p" podUID="c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7" Nov 02 21:08:04.766372 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:04.766335 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:08:04.766416 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:04.766393 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s98xl" podUID="d45d9164-2ba4-400d-8b27-273d7f4f351d" Nov 02 21:08:04.766484 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:04.766463 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bllxh" podUID="87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1" Nov 02 21:08:05.956070 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:05.956036 2577 generic.go:356] "Generic (PLEG): container finished" podID="7dab1353-88e8-4250-b7ab-a4b2a69e37fa" containerID="df1cde4b7b09523fc50163615d6b83908fb0bc65988da58df5899ec5de7c311a" exitCode=0 Nov 02 21:08:05.956724 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:05.956125 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94zch" event={"ID":"7dab1353-88e8-4250-b7ab-a4b2a69e37fa","Type":"ContainerDied","Data":"df1cde4b7b09523fc50163615d6b83908fb0bc65988da58df5899ec5de7c311a"} Nov 02 21:08:05.959030 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:05.959012 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 21:08:05.959365 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:05.959346 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" event={"ID":"bff8d097-26a3-4621-ac01-c3b00ed23e06","Type":"ContainerStarted","Data":"3d36343147ac675622cd5cf045560ead46696128cc227f316cc5a6a3d07737c5"} Nov 02 21:08:06.194698 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:06.194665 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mz5qm" Nov 02 21:08:06.194870 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:06.194802 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 21:08:06.195229 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:06.195203 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mz5qm" Nov 02 21:08:06.766328 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:06.766298 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:08:06.766478 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:06.766298 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:08:06.766478 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:06.766405 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s98xl" podUID="d45d9164-2ba4-400d-8b27-273d7f4f351d" Nov 02 21:08:06.766478 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:06.766298 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:08:06.766600 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:06.766481 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c67p" podUID="c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7" Nov 02 21:08:06.766600 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:06.766536 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bllxh" podUID="87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1" Nov 02 21:08:06.964560 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:06.964529 2577 generic.go:356] "Generic (PLEG): container finished" podID="7dab1353-88e8-4250-b7ab-a4b2a69e37fa" containerID="f1fae03813ba6dabd2a6fd55d4c5331ec3dd74d39091576e17bc1489c5ec7259" exitCode=0 Nov 02 21:08:06.964908 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:06.964616 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94zch" event={"ID":"7dab1353-88e8-4250-b7ab-a4b2a69e37fa","Type":"ContainerDied","Data":"f1fae03813ba6dabd2a6fd55d4c5331ec3dd74d39091576e17bc1489c5ec7259"} Nov 02 21:08:07.970356 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:07.970328 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 21:08:07.970990 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:07.970724 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" event={"ID":"bff8d097-26a3-4621-ac01-c3b00ed23e06","Type":"ContainerStarted","Data":"ebdab94f01cd6546e1d9715d8af91223cf4249b70731056f920f34123d403fff"} Nov 02 21:08:07.970990 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:07.970975 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:07.971105 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:07.971066 2577 scope.go:122] "RemoveContainer" containerID="c6cb2944006d1ee69c92ed1bb854345a65049bfff5f2cf4a817992c7eb338b8e" Nov 02 21:08:07.974621 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:07.974588 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94zch" event={"ID":"7dab1353-88e8-4250-b7ab-a4b2a69e37fa","Type":"ContainerStarted","Data":"04c8aa62aea0122f90c30daeae63d3716de12a6864b57ead522b1adf3beabaca"} Nov 02 21:08:07.988011 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:07.987985 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:08.305579 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:08.305527 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-94zch" podStartSLOduration=9.568753829 podStartE2EDuration="23.305513473s" totalImagesPullingTime="13.736759644s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:07:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:08.305487967 +0000 UTC m=+22.962562856" watchObservedRunningTime="2026-11-02 21:08:08.305513473 +0000 UTC m=+22.962588356" Nov 02 21:08:08.766751 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:08.766724 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:08:08.766931 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:08.766821 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:08:08.766990 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:08.766926 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:08:08.766990 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:08.766946 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bllxh" podUID="87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1" Nov 02 21:08:08.767082 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:08.767060 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s98xl" podUID="d45d9164-2ba4-400d-8b27-273d7f4f351d" Nov 02 21:08:08.767166 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:08.767119 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c67p" podUID="c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7" Nov 02 21:08:08.980215 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:08.980186 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 21:08:08.980676 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:08.980575 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" event={"ID":"bff8d097-26a3-4621-ac01-c3b00ed23e06","Type":"ContainerStarted","Data":"19aaa5f775ee106d255bff7628403a66f4e24713a78bfdf4f73641aac62d82bf"} Nov 02 21:08:08.981026 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:08.981006 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:08.981103 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:08.981088 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:08.995586 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:08.995560 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:09.282905 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:09.282857 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" podStartSLOduration=8.166823287 podStartE2EDuration="23.282843478s" totalImagesPullingTime="15.116020191s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:09.158494892 +0000 UTC m=+23.815569780" watchObservedRunningTime="2026-11-02 21:08:09.282843478 +0000 UTC m=+23.939918366" Nov 02 21:08:09.953840 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:09.953805 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bllxh"] Nov 02 21:08:09.953991 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:09.953915 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:08:09.954028 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:09.954003 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bllxh" podUID="87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1" Nov 02 21:08:09.954393 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:09.954374 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8c67p"] Nov 02 21:08:09.954468 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:09.954457 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:08:09.954542 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:09.954526 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c67p" podUID="c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7" Nov 02 21:08:09.957015 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:09.956998 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s98xl"] Nov 02 21:08:09.957080 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:09.957070 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:08:09.957169 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:09.957156 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s98xl" podUID="d45d9164-2ba4-400d-8b27-273d7f4f351d" Nov 02 21:08:11.766581 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:11.766552 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:08:11.767021 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:11.766588 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:08:11.767021 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:11.766676 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:08:11.767021 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:11.766788 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c67p" podUID="c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7" Nov 02 21:08:11.767021 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:11.766905 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bllxh" podUID="87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1" Nov 02 21:08:11.767186 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:11.767046 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s98xl" podUID="d45d9164-2ba4-400d-8b27-273d7f4f351d" Nov 02 21:08:13.765862 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:13.765828 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:08:13.765862 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:13.765839 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:08:13.766478 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:13.766000 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c67p" podUID="c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7" Nov 02 21:08:13.766478 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:13.766031 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:08:13.766478 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:13.766096 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bllxh" podUID="87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1" Nov 02 21:08:13.766478 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:13.766205 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s98xl" podUID="d45d9164-2ba4-400d-8b27-273d7f4f351d" Nov 02 21:08:15.767182 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:15.766982 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:08:15.767653 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:15.767024 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:08:15.767653 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:15.767253 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bllxh" podUID="87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1" Nov 02 21:08:15.767653 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:15.767080 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:08:15.767653 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:15.767327 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8c67p" podUID="c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7" Nov 02 21:08:15.767653 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:15.767430 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s98xl" podUID="d45d9164-2ba4-400d-8b27-273d7f4f351d" Nov 02 21:08:16.089361 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.089333 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-89.us-west-2.compute.internal" event="NodeReady" Nov 02 21:08:16.089511 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.089458 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 21:08:16.204384 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.204351 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nv2sw"] Nov 02 21:08:16.206199 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.206178 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5zc9z"] Nov 02 21:08:16.206310 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.206287 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-nv2sw" Nov 02 21:08:16.207728 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.207707 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bb7st"] Nov 02 21:08:16.207871 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.207849 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5zc9z" Nov 02 21:08:16.209305 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.209289 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bb7st" Nov 02 21:08:16.247355 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.247323 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c987dc54-2p2h6"] Nov 02 21:08:16.252991 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.252964 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf"] Nov 02 21:08:16.254309 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.254283 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:16.258821 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.258798 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6"] Nov 02 21:08:16.258935 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.258919 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf" Nov 02 21:08:16.263990 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.263971 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qpf24"] Nov 02 21:08:16.264116 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.264099 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6" Nov 02 21:08:16.278922 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.278880 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-2ftg6"] Nov 02 21:08:16.279239 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.279016 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qpf24" Nov 02 21:08:16.287493 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.287467 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-798b7b6b45-hlcl7"] Nov 02 21:08:16.287592 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.287571 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:16.295758 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.295738 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq"] Nov 02 21:08:16.295859 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.295842 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:16.302627 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.302417 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 02 21:08:16.302627 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.302539 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 02 21:08:16.305366 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.305349 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-pfjlk"] Nov 02 21:08:16.305494 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.305472 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" Nov 02 21:08:16.309766 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.309742 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 21:08:16.309939 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.309880 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 21:08:16.313539 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.313487 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qshj9\"" Nov 02 21:08:16.314886 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.314856 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-qd9n6"] Nov 02 21:08:16.315016 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.314987 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pfjlk" Nov 02 21:08:16.327491 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.327468 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cf9df1a4-3684-464c-97f3-21a3151a7183-image-registry-private-configuration\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:16.327604 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.327496 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ca28f2b-c77d-4aa1-8501-ec380253ab88-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf\" (UID: \"5ca28f2b-c77d-4aa1-8501-ec380253ab88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf" Nov 02 21:08:16.327604 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.327529 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgxs2\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-kube-api-access-kgxs2\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:16.327670 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.327627 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:16.327670 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.327653 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cf9df1a4-3684-464c-97f3-21a3151a7183-trusted-ca\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:16.327733 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.327674 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gttn\" (UniqueName: \"kubernetes.io/projected/a899c9c5-79b4-472d-942c-5e0c4c53e954-kube-api-access-9gttn\") pod \"cluster-monitoring-operator-884576c57-z4kx6\" (UID: \"a899c9c5-79b4-472d-942c-5e0c4c53e954\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6" Nov 02 21:08:16.327733 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.327696 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfhs6\" (UniqueName: \"kubernetes.io/projected/345370c7-0ca8-4afe-94db-c9f5394d4d72-kube-api-access-rfhs6\") pod \"kas-connection-checker-685895c7c9-nv2sw\" (UID: \"345370c7-0ca8-4afe-94db-c9f5394d4d72\") " pod="kube-system/kas-connection-checker-685895c7c9-nv2sw" Nov 02 21:08:16.327811 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.327789 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cf9df1a4-3684-464c-97f3-21a3151a7183-installation-pull-secrets\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:16.327878 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.327864 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cf9df1a4-3684-464c-97f3-21a3151a7183-ca-trust-extracted\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:16.327912 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.327886 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-certificates\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:16.327943 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.327907 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7kkw\" (UniqueName: \"kubernetes.io/projected/5ca28f2b-c77d-4aa1-8501-ec380253ab88-kube-api-access-t7kkw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf\" (UID: \"5ca28f2b-c77d-4aa1-8501-ec380253ab88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf" Nov 02 21:08:16.327943 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.327936 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a899c9c5-79b4-472d-942c-5e0c4c53e954-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-z4kx6\" (UID: \"a899c9c5-79b4-472d-942c-5e0c4c53e954\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6" Nov 02 21:08:16.328001 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.327967 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ca28f2b-c77d-4aa1-8501-ec380253ab88-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf\" (UID: \"5ca28f2b-c77d-4aa1-8501-ec380253ab88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf" Nov 02 21:08:16.328001 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.327989 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5sx2\" (UniqueName: \"kubernetes.io/projected/d3acf985-6085-4061-8ca8-e6d7036cb6fd-kube-api-access-x5sx2\") pod \"kas-connection-checker-685895c7c9-5zc9z\" (UID: \"d3acf985-6085-4061-8ca8-e6d7036cb6fd\") " pod="kube-system/kas-connection-checker-685895c7c9-5zc9z" Nov 02 21:08:16.328061 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.328009 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv27q\" (UniqueName: \"kubernetes.io/projected/c4a07e29-b928-4427-b4c2-5315984f03b3-kube-api-access-qv27q\") pod \"kas-connection-checker-685895c7c9-bb7st\" (UID: \"c4a07e29-b928-4427-b4c2-5315984f03b3\") " pod="kube-system/kas-connection-checker-685895c7c9-bb7st" Nov 02 21:08:16.328061 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.328043 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-bound-sa-token\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:16.328119 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.328090 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-z4kx6\" (UID: \"a899c9c5-79b4-472d-942c-5e0c4c53e954\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6" Nov 02 21:08:16.328466 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.328454 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-588654849f-w89ht"] Nov 02 21:08:16.328611 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.328593 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:16.339538 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.339480 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-qpklq"] Nov 02 21:08:16.339659 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.339597 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:16.347332 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.347312 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-8jclh"] Nov 02 21:08:16.347448 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.347432 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-qpklq" Nov 02 21:08:16.354146 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.354110 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nv2sw"] Nov 02 21:08:16.354235 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.354167 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bb7st"] Nov 02 21:08:16.354235 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.354188 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5zc9z"] Nov 02 21:08:16.354235 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.354210 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vn62q"] Nov 02 21:08:16.354388 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.354374 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8jclh" Nov 02 21:08:16.359149 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.359111 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mlgzw"] Nov 02 21:08:16.361042 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.360527 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:16.364592 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.364570 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 21:08:16.364695 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.364599 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 02 21:08:16.364927 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.364702 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 02 21:08:16.364998 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.364717 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 21:08:16.364998 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.364766 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 21:08:16.364998 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.364778 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:08:16.364998 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.364834 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 21:08:16.364998 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.364841 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 21:08:16.364998 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.364855 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 02 21:08:16.364998 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.364993 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 02 21:08:16.364998 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.364855 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:08:16.364998 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.364866 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 21:08:16.365422 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.364884 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 02 21:08:16.365422 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.364901 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 21:08:16.365422 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.365073 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 02 21:08:16.365422 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.365073 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:08:16.367586 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.367570 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-2ftg6"] Nov 02 21:08:16.367640 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.367599 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6"] Nov 02 21:08:16.367640 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.367615 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf"] Nov 02 21:08:16.367640 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.367633 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq"] Nov 02 21:08:16.367724 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.367650 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c987dc54-2p2h6"] Nov 02 21:08:16.367724 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.367667 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-qd9n6"] Nov 02 21:08:16.367724 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.367683 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-qpklq"] Nov 02 21:08:16.367724 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.367696 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-8jclh"] Nov 02 21:08:16.367724 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.367709 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-588654849f-w89ht"] Nov 02 21:08:16.367724 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.367724 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vn62q"] Nov 02 21:08:16.367880 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.367741 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qpf24"] Nov 02 21:08:16.367880 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.367754 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-pfjlk"] Nov 02 21:08:16.367880 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.367767 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-798b7b6b45-hlcl7"] Nov 02 21:08:16.367880 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.367707 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mlgzw" Nov 02 21:08:16.371527 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.371510 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 02 21:08:16.417236 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.417208 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:08:16.417385 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.417297 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 02 21:08:16.417385 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.417352 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 02 21:08:16.417385 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.417370 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 02 21:08:16.417385 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.417374 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 21:08:16.417385 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.417378 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 02 21:08:16.417634 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.417353 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 02 21:08:16.417634 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.417414 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 02 21:08:16.417634 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.417458 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 02 21:08:16.417634 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.417507 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 02 21:08:16.417634 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.417625 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-sntml\"" Nov 02 21:08:16.417863 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.417649 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 02 21:08:16.417863 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.417654 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-ml487\"" Nov 02 21:08:16.417863 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.417684 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 02 21:08:16.417863 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.417717 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 02 21:08:16.417863 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.417770 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 21:08:16.417863 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.417722 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-r64zg\"" Nov 02 21:08:16.417863 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.417786 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-t7psx\"" Nov 02 21:08:16.417863 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.417630 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 02 21:08:16.417863 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.417826 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ftl5q\"" Nov 02 21:08:16.417863 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.417725 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 21:08:16.418294 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.417920 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-58fxm\"" Nov 02 21:08:16.418294 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.418065 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 21:08:16.418294 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.418085 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-97nlp\"" Nov 02 21:08:16.418294 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.418193 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vvdb6\"" Nov 02 21:08:16.418294 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.418258 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-2pl68\"" Nov 02 21:08:16.418569 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.418552 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-5jdjp\"" Nov 02 21:08:16.418634 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.418584 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:16.418690 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.418611 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-m78tx\"" Nov 02 21:08:16.419416 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.419400 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mlgzw"] Nov 02 21:08:16.421253 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.421241 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 21:08:16.421289 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.421259 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-d9psf\"" Nov 02 21:08:16.428497 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.428481 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-stats-auth\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:16.428544 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.428511 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cf9df1a4-3684-464c-97f3-21a3151a7183-trusted-ca\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:16.428544 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.428529 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v2wr\" (UniqueName: \"kubernetes.io/projected/8cf65e09-2079-40cc-b597-1b67ba482a04-kube-api-access-5v2wr\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:16.428620 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.428554 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f264bf41-80c4-4dda-a67e-6651b464655c-serving-cert\") pod \"service-ca-operator-544d749c8c-pfjlk\" (UID: \"f264bf41-80c4-4dda-a67e-6651b464655c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pfjlk" Nov 02 21:08:16.428702 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.428688 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:16.428749 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.428717 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-bound-sa-token\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:16.428810 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.428790 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a2820a3-6d1e-42d0-8a89-0e47f870abc4-trusted-ca\") pod \"console-operator-7cd56b8dbc-qd9n6\" (UID: \"9a2820a3-6d1e-42d0-8a89-0e47f870abc4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:16.428867 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.428848 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08b38b87-8fd7-475e-94db-8bdd1339dc4b-trusted-ca\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:16.428920 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.428887 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ad2a39a-7a74-44a2-a97f-88e84e65616c-tmp-dir\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:16.428968 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.428924 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7kkw\" (UniqueName: \"kubernetes.io/projected/5ca28f2b-c77d-4aa1-8501-ec380253ab88-kube-api-access-t7kkw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf\" (UID: \"5ca28f2b-c77d-4aa1-8501-ec380253ab88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf" Nov 02 21:08:16.428968 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.428952 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a899c9c5-79b4-472d-942c-5e0c4c53e954-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-z4kx6\" (UID: \"a899c9c5-79b4-472d-942c-5e0c4c53e954\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6" Nov 02 21:08:16.429049 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.429010 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-default-certificate\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:16.429101 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.429048 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx9qw\" (UniqueName: \"kubernetes.io/projected/934bd6cf-3dcb-40fd-9b11-97886b3f098b-kube-api-access-vx9qw\") pod \"volume-data-source-validator-58c9f5f46d-qpf24\" (UID: \"934bd6cf-3dcb-40fd-9b11-97886b3f098b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qpf24" Nov 02 21:08:16.429176 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.429107 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/24c2ea1f-b61b-4e62-914a-2c52508bc04d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-qpklq\" (UID: \"24c2ea1f-b61b-4e62-914a-2c52508bc04d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qpklq" Nov 02 21:08:16.429176 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.429156 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxm2l\" (UniqueName: \"kubernetes.io/projected/7de0fc42-d787-49e7-80a2-5122a67bfba0-kube-api-access-jxm2l\") pod \"ingress-canary-mlgzw\" (UID: \"7de0fc42-d787-49e7-80a2-5122a67bfba0\") " pod="openshift-ingress-canary/ingress-canary-mlgzw" Nov 02 21:08:16.429286 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.429184 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vmqkq\" (UID: \"6b5655ce-43ca-48e4-9e4b-872283ccb85c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" Nov 02 21:08:16.429286 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.429242 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-bound-sa-token\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:16.429379 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.429315 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-z4kx6\" (UID: \"a899c9c5-79b4-472d-942c-5e0c4c53e954\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6" Nov 02 21:08:16.429428 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:16.429380 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 21:08:16.429428 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.429389 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08b38b87-8fd7-475e-94db-8bdd1339dc4b-image-registry-private-configuration\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:16.429428 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.429417 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxcnw\" (UniqueName: \"kubernetes.io/projected/44a209b7-2144-45a0-bea0-2f8adf6658cb-kube-api-access-hxcnw\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:16.429428 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:16.429425 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls podName:a899c9c5-79b4-472d-942c-5e0c4c53e954 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:16.929412196 +0000 UTC m=+31.586487064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-z4kx6" (UID: "a899c9c5-79b4-472d-942c-5e0c4c53e954") : secret "cluster-monitoring-operator-tls" not found Nov 02 21:08:16.429592 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.429471 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vmqkq\" (UID: \"6b5655ce-43ca-48e4-9e4b-872283ccb85c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" Nov 02 21:08:16.429592 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.429510 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cf9df1a4-3684-464c-97f3-21a3151a7183-trusted-ca\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:16.429592 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.429517 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cf65e09-2079-40cc-b597-1b67ba482a04-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:16.429592 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.429556 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cf65e09-2079-40cc-b597-1b67ba482a04-service-ca-bundle\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:16.429711 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.429596 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a899c9c5-79b4-472d-942c-5e0c4c53e954-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-z4kx6\" (UID: \"a899c9c5-79b4-472d-942c-5e0c4c53e954\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6" Nov 02 21:08:16.429711 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.429608 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f264bf41-80c4-4dda-a67e-6651b464655c-config\") pod \"service-ca-operator-544d749c8c-pfjlk\" (UID: \"f264bf41-80c4-4dda-a67e-6651b464655c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pfjlk" Nov 02 21:08:16.429711 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.429641 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgxs2\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-kube-api-access-kgxs2\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:16.429711 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.429671 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-certificates\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:16.429819 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.429708 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cf9df1a4-3684-464c-97f3-21a3151a7183-image-registry-private-configuration\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:16.429819 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.429741 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ca28f2b-c77d-4aa1-8501-ec380253ab88-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf\" (UID: \"5ca28f2b-c77d-4aa1-8501-ec380253ab88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf" Nov 02 21:08:16.429819 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.429768 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad2a39a-7a74-44a2-a97f-88e84e65616c-config-volume\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:16.429819 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.429805 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:16.429935 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:16.429873 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:08:16.429935 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:16.429887 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c987dc54-2p2h6: secret "image-registry-tls" not found Nov 02 21:08:16.429935 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:16.429919 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls podName:cf9df1a4-3684-464c-97f3-21a3151a7183 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:16.929908936 +0000 UTC m=+31.586983839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls") pod "image-registry-c987dc54-2p2h6" (UID: "cf9df1a4-3684-464c-97f3-21a3151a7183") : secret "image-registry-tls" not found Nov 02 21:08:16.429935 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.429914 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9a2820a3-6d1e-42d0-8a89-0e47f870abc4-serving-cert\") pod \"console-operator-7cd56b8dbc-qd9n6\" (UID: \"9a2820a3-6d1e-42d0-8a89-0e47f870abc4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:16.430123 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.429947 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxbh5\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-kube-api-access-bxbh5\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:16.430123 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.429999 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8cf65e09-2079-40cc-b597-1b67ba482a04-serving-cert\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:16.430123 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.430035 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08b38b87-8fd7-475e-94db-8bdd1339dc4b-ca-trust-extracted\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:16.430123 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.430067 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gttn\" (UniqueName: \"kubernetes.io/projected/a899c9c5-79b4-472d-942c-5e0c4c53e954-kube-api-access-9gttn\") pod \"cluster-monitoring-operator-884576c57-z4kx6\" (UID: \"a899c9c5-79b4-472d-942c-5e0c4c53e954\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6" Nov 02 21:08:16.430331 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.430123 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfhs6\" (UniqueName: \"kubernetes.io/projected/345370c7-0ca8-4afe-94db-c9f5394d4d72-kube-api-access-rfhs6\") pod \"kas-connection-checker-685895c7c9-nv2sw\" (UID: \"345370c7-0ca8-4afe-94db-c9f5394d4d72\") " pod="kube-system/kas-connection-checker-685895c7c9-nv2sw" Nov 02 21:08:16.430331 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.430172 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmqww\" (UniqueName: \"kubernetes.io/projected/4ad2a39a-7a74-44a2-a97f-88e84e65616c-kube-api-access-fmqww\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:16.430331 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.430201 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cf9df1a4-3684-464c-97f3-21a3151a7183-installation-pull-secrets\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:16.430331 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.430244 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cf65e09-2079-40cc-b597-1b67ba482a04-tmp\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:16.430331 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.430262 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ca28f2b-c77d-4aa1-8501-ec380253ab88-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf\" (UID: \"5ca28f2b-c77d-4aa1-8501-ec380253ab88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf" Nov 02 21:08:16.430331 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.430301 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cf9df1a4-3684-464c-97f3-21a3151a7183-ca-trust-extracted\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:16.430331 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.430330 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-certificates\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:16.430651 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.430363 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qpklq\" (UID: \"24c2ea1f-b61b-4e62-914a-2c52508bc04d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qpklq" Nov 02 21:08:16.430651 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.430394 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw9lp\" (UniqueName: \"kubernetes.io/projected/f264bf41-80c4-4dda-a67e-6651b464655c-kube-api-access-gw9lp\") pod \"service-ca-operator-544d749c8c-pfjlk\" (UID: \"f264bf41-80c4-4dda-a67e-6651b464655c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pfjlk" Nov 02 21:08:16.430651 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.430502 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnvnx\" (UniqueName: \"kubernetes.io/projected/9a2820a3-6d1e-42d0-8a89-0e47f870abc4-kube-api-access-gnvnx\") pod \"console-operator-7cd56b8dbc-qd9n6\" (UID: \"9a2820a3-6d1e-42d0-8a89-0e47f870abc4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:16.430651 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.430535 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w7df\" (UniqueName: \"kubernetes.io/projected/fecc158a-a288-4064-b3e9-6fe91e703f5b-kube-api-access-7w7df\") pod \"network-check-source-6f78d59f99-8jclh\" (UID: \"fecc158a-a288-4064-b3e9-6fe91e703f5b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8jclh" Nov 02 21:08:16.430651 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.430567 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ca28f2b-c77d-4aa1-8501-ec380253ab88-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf\" (UID: \"5ca28f2b-c77d-4aa1-8501-ec380253ab88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf" Nov 02 21:08:16.430651 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.430607 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5sx2\" (UniqueName: \"kubernetes.io/projected/d3acf985-6085-4061-8ca8-e6d7036cb6fd-kube-api-access-x5sx2\") pod \"kas-connection-checker-685895c7c9-5zc9z\" (UID: \"d3acf985-6085-4061-8ca8-e6d7036cb6fd\") " pod="kube-system/kas-connection-checker-685895c7c9-5zc9z" Nov 02 21:08:16.430651 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.430635 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv27q\" (UniqueName: \"kubernetes.io/projected/c4a07e29-b928-4427-b4c2-5315984f03b3-kube-api-access-qv27q\") pod \"kas-connection-checker-685895c7c9-bb7st\" (UID: \"c4a07e29-b928-4427-b4c2-5315984f03b3\") " pod="kube-system/kas-connection-checker-685895c7c9-bb7st" Nov 02 21:08:16.430967 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.430659 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08b38b87-8fd7-475e-94db-8bdd1339dc4b-installation-pull-secrets\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:16.430967 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.430667 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cf9df1a4-3684-464c-97f3-21a3151a7183-ca-trust-extracted\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:16.430967 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.430694 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8cf65e09-2079-40cc-b597-1b67ba482a04-snapshots\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:16.430967 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.430718 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:16.430967 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.430742 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:16.430967 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.430772 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7de0fc42-d787-49e7-80a2-5122a67bfba0-cert\") pod \"ingress-canary-mlgzw\" (UID: \"7de0fc42-d787-49e7-80a2-5122a67bfba0\") " pod="openshift-ingress-canary/ingress-canary-mlgzw" Nov 02 21:08:16.430967 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.430795 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q672k\" (UniqueName: \"kubernetes.io/projected/6b5655ce-43ca-48e4-9e4b-872283ccb85c-kube-api-access-q672k\") pod \"cluster-samples-operator-6ddb6c8b94-vmqkq\" (UID: \"6b5655ce-43ca-48e4-9e4b-872283ccb85c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" Nov 02 21:08:16.430967 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.430819 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9a2820a3-6d1e-42d0-8a89-0e47f870abc4-config\") pod \"console-operator-7cd56b8dbc-qd9n6\" (UID: \"9a2820a3-6d1e-42d0-8a89-0e47f870abc4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:16.430967 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.430827 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-certificates\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:16.430967 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.430838 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:16.433738 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.433712 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ca28f2b-c77d-4aa1-8501-ec380253ab88-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf\" (UID: \"5ca28f2b-c77d-4aa1-8501-ec380253ab88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf" Nov 02 21:08:16.433827 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.433724 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cf9df1a4-3684-464c-97f3-21a3151a7183-image-registry-private-configuration\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:16.433865 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.433827 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cf9df1a4-3684-464c-97f3-21a3151a7183-installation-pull-secrets\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:16.465085 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.465061 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 02 21:08:16.465275 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.465257 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:08:16.465326 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.465303 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 21:08:16.465326 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.465319 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 21:08:16.465431 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.465326 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 02 21:08:16.465491 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.465471 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 21:08:16.465491 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.465475 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 21:08:16.465572 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.465525 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 02 21:08:16.465572 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.465563 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 21:08:16.466301 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.466287 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 02 21:08:16.488571 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.488540 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7kkw\" (UniqueName: \"kubernetes.io/projected/5ca28f2b-c77d-4aa1-8501-ec380253ab88-kube-api-access-t7kkw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf\" (UID: \"5ca28f2b-c77d-4aa1-8501-ec380253ab88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf" Nov 02 21:08:16.488671 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.488607 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgxs2\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-kube-api-access-kgxs2\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:16.488955 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.488923 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-bound-sa-token\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:16.489030 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.488951 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5sx2\" (UniqueName: \"kubernetes.io/projected/d3acf985-6085-4061-8ca8-e6d7036cb6fd-kube-api-access-x5sx2\") pod \"kas-connection-checker-685895c7c9-5zc9z\" (UID: \"d3acf985-6085-4061-8ca8-e6d7036cb6fd\") " pod="kube-system/kas-connection-checker-685895c7c9-5zc9z" Nov 02 21:08:16.489628 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.489610 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv27q\" (UniqueName: \"kubernetes.io/projected/c4a07e29-b928-4427-b4c2-5315984f03b3-kube-api-access-qv27q\") pod \"kas-connection-checker-685895c7c9-bb7st\" (UID: \"c4a07e29-b928-4427-b4c2-5315984f03b3\") " pod="kube-system/kas-connection-checker-685895c7c9-bb7st" Nov 02 21:08:16.489701 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.489638 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gttn\" (UniqueName: \"kubernetes.io/projected/a899c9c5-79b4-472d-942c-5e0c4c53e954-kube-api-access-9gttn\") pod \"cluster-monitoring-operator-884576c57-z4kx6\" (UID: \"a899c9c5-79b4-472d-942c-5e0c4c53e954\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6" Nov 02 21:08:16.489701 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.489676 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfhs6\" (UniqueName: \"kubernetes.io/projected/345370c7-0ca8-4afe-94db-c9f5394d4d72-kube-api-access-rfhs6\") pod \"kas-connection-checker-685895c7c9-nv2sw\" (UID: \"345370c7-0ca8-4afe-94db-c9f5394d4d72\") " pod="kube-system/kas-connection-checker-685895c7c9-nv2sw" Nov 02 21:08:16.522535 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.522511 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-nv2sw" Nov 02 21:08:16.524646 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.524628 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5zc9z" Nov 02 21:08:16.531926 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.531907 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q672k\" (UniqueName: \"kubernetes.io/projected/6b5655ce-43ca-48e4-9e4b-872283ccb85c-kube-api-access-q672k\") pod \"cluster-samples-operator-6ddb6c8b94-vmqkq\" (UID: \"6b5655ce-43ca-48e4-9e4b-872283ccb85c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" Nov 02 21:08:16.532043 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.531936 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9a2820a3-6d1e-42d0-8a89-0e47f870abc4-config\") pod \"console-operator-7cd56b8dbc-qd9n6\" (UID: \"9a2820a3-6d1e-42d0-8a89-0e47f870abc4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:16.532043 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.531955 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:16.532043 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.531974 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-stats-auth\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:16.532043 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532008 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v2wr\" (UniqueName: \"kubernetes.io/projected/8cf65e09-2079-40cc-b597-1b67ba482a04-kube-api-access-5v2wr\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:16.532043 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532034 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f264bf41-80c4-4dda-a67e-6651b464655c-serving-cert\") pod \"service-ca-operator-544d749c8c-pfjlk\" (UID: \"f264bf41-80c4-4dda-a67e-6651b464655c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pfjlk" Nov 02 21:08:16.532330 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532060 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:16.532330 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:16.532152 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:08:16.532330 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:16.532153 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle podName:44a209b7-2144-45a0-bea0-2f8adf6658cb nodeName:}" failed. No retries permitted until 2026-11-02 21:08:17.032112927 +0000 UTC m=+31.689187815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle") pod "router-default-798b7b6b45-hlcl7" (UID: "44a209b7-2144-45a0-bea0-2f8adf6658cb") : configmap references non-existent config key: service-ca.crt Nov 02 21:08:16.532330 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532208 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-bound-sa-token\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:16.532330 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532238 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a2820a3-6d1e-42d0-8a89-0e47f870abc4-trusted-ca\") pod \"console-operator-7cd56b8dbc-qd9n6\" (UID: \"9a2820a3-6d1e-42d0-8a89-0e47f870abc4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:16.532330 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532265 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08b38b87-8fd7-475e-94db-8bdd1339dc4b-trusted-ca\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:16.532330 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532289 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ad2a39a-7a74-44a2-a97f-88e84e65616c-tmp-dir\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:16.532330 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532300 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bb7st" Nov 02 21:08:16.532330 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532315 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-default-certificate\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:16.532809 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532340 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx9qw\" (UniqueName: \"kubernetes.io/projected/934bd6cf-3dcb-40fd-9b11-97886b3f098b-kube-api-access-vx9qw\") pod \"volume-data-source-validator-58c9f5f46d-qpf24\" (UID: \"934bd6cf-3dcb-40fd-9b11-97886b3f098b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qpf24" Nov 02 21:08:16.532809 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:16.532362 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls podName:4ad2a39a-7a74-44a2-a97f-88e84e65616c nodeName:}" failed. No retries permitted until 2026-11-02 21:08:17.032343336 +0000 UTC m=+31.689418207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls") pod "dns-default-vn62q" (UID: "4ad2a39a-7a74-44a2-a97f-88e84e65616c") : secret "dns-default-metrics-tls" not found Nov 02 21:08:16.532809 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532402 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/24c2ea1f-b61b-4e62-914a-2c52508bc04d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-qpklq\" (UID: \"24c2ea1f-b61b-4e62-914a-2c52508bc04d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qpklq" Nov 02 21:08:16.532809 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532434 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxm2l\" (UniqueName: \"kubernetes.io/projected/7de0fc42-d787-49e7-80a2-5122a67bfba0-kube-api-access-jxm2l\") pod \"ingress-canary-mlgzw\" (UID: \"7de0fc42-d787-49e7-80a2-5122a67bfba0\") " pod="openshift-ingress-canary/ingress-canary-mlgzw" Nov 02 21:08:16.532809 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532470 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vmqkq\" (UID: \"6b5655ce-43ca-48e4-9e4b-872283ccb85c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" Nov 02 21:08:16.532809 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532512 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08b38b87-8fd7-475e-94db-8bdd1339dc4b-image-registry-private-configuration\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:16.532809 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532536 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxcnw\" (UniqueName: \"kubernetes.io/projected/44a209b7-2144-45a0-bea0-2f8adf6658cb-kube-api-access-hxcnw\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:16.532809 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532565 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vmqkq\" (UID: \"6b5655ce-43ca-48e4-9e4b-872283ccb85c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" Nov 02 21:08:16.532809 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532589 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cf65e09-2079-40cc-b597-1b67ba482a04-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:16.532809 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532615 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cf65e09-2079-40cc-b597-1b67ba482a04-service-ca-bundle\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:16.532809 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532646 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f264bf41-80c4-4dda-a67e-6651b464655c-config\") pod \"service-ca-operator-544d749c8c-pfjlk\" (UID: \"f264bf41-80c4-4dda-a67e-6651b464655c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pfjlk" Nov 02 21:08:16.532809 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532685 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-certificates\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:16.532809 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532689 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ad2a39a-7a74-44a2-a97f-88e84e65616c-tmp-dir\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:16.532809 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532732 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad2a39a-7a74-44a2-a97f-88e84e65616c-config-volume\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:16.532809 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532736 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9a2820a3-6d1e-42d0-8a89-0e47f870abc4-config\") pod \"console-operator-7cd56b8dbc-qd9n6\" (UID: \"9a2820a3-6d1e-42d0-8a89-0e47f870abc4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:16.533553 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532804 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9a2820a3-6d1e-42d0-8a89-0e47f870abc4-serving-cert\") pod \"console-operator-7cd56b8dbc-qd9n6\" (UID: \"9a2820a3-6d1e-42d0-8a89-0e47f870abc4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:16.533553 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532835 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxbh5\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-kube-api-access-bxbh5\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:16.533553 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:16.532850 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 21:08:16.533553 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532864 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8cf65e09-2079-40cc-b597-1b67ba482a04-serving-cert\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:16.533553 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:16.532896 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls podName:6b5655ce-43ca-48e4-9e4b-872283ccb85c nodeName:}" failed. No retries permitted until 2026-11-02 21:08:17.03288301 +0000 UTC m=+31.689957917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vmqkq" (UID: "6b5655ce-43ca-48e4-9e4b-872283ccb85c") : secret "samples-operator-tls" not found Nov 02 21:08:16.533553 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532918 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08b38b87-8fd7-475e-94db-8bdd1339dc4b-ca-trust-extracted\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:16.533553 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532957 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmqww\" (UniqueName: \"kubernetes.io/projected/4ad2a39a-7a74-44a2-a97f-88e84e65616c-kube-api-access-fmqww\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:16.533553 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.532988 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cf65e09-2079-40cc-b597-1b67ba482a04-tmp\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:16.533553 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.533025 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qpklq\" (UID: \"24c2ea1f-b61b-4e62-914a-2c52508bc04d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qpklq" Nov 02 21:08:16.533553 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.533054 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw9lp\" (UniqueName: \"kubernetes.io/projected/f264bf41-80c4-4dda-a67e-6651b464655c-kube-api-access-gw9lp\") pod \"service-ca-operator-544d749c8c-pfjlk\" (UID: \"f264bf41-80c4-4dda-a67e-6651b464655c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pfjlk" Nov 02 21:08:16.533553 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.533089 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnvnx\" (UniqueName: \"kubernetes.io/projected/9a2820a3-6d1e-42d0-8a89-0e47f870abc4-kube-api-access-gnvnx\") pod \"console-operator-7cd56b8dbc-qd9n6\" (UID: \"9a2820a3-6d1e-42d0-8a89-0e47f870abc4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:16.533553 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.533118 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w7df\" (UniqueName: \"kubernetes.io/projected/fecc158a-a288-4064-b3e9-6fe91e703f5b-kube-api-access-7w7df\") pod \"network-check-source-6f78d59f99-8jclh\" (UID: \"fecc158a-a288-4064-b3e9-6fe91e703f5b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8jclh" Nov 02 21:08:16.533553 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.533180 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08b38b87-8fd7-475e-94db-8bdd1339dc4b-installation-pull-secrets\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:16.533553 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.533221 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8cf65e09-2079-40cc-b597-1b67ba482a04-snapshots\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:16.533553 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.533246 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:16.533553 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.533263 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/24c2ea1f-b61b-4e62-914a-2c52508bc04d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-qpklq\" (UID: \"24c2ea1f-b61b-4e62-914a-2c52508bc04d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qpklq" Nov 02 21:08:16.534376 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:16.533352 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 21:08:16.534376 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.533366 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08b38b87-8fd7-475e-94db-8bdd1339dc4b-trusted-ca\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:16.534376 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:16.533393 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs podName:44a209b7-2144-45a0-bea0-2f8adf6658cb nodeName:}" failed. No retries permitted until 2026-11-02 21:08:17.033380504 +0000 UTC m=+31.690455386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs") pod "router-default-798b7b6b45-hlcl7" (UID: "44a209b7-2144-45a0-bea0-2f8adf6658cb") : secret "router-metrics-certs-default" not found Nov 02 21:08:16.534376 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.533270 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:16.534376 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.533439 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7de0fc42-d787-49e7-80a2-5122a67bfba0-cert\") pod \"ingress-canary-mlgzw\" (UID: \"7de0fc42-d787-49e7-80a2-5122a67bfba0\") " pod="openshift-ingress-canary/ingress-canary-mlgzw" Nov 02 21:08:16.534376 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.533780 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vmqkq\" (UID: \"6b5655ce-43ca-48e4-9e4b-872283ccb85c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" Nov 02 21:08:16.534376 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.533844 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cf65e09-2079-40cc-b597-1b67ba482a04-service-ca-bundle\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:16.534376 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.534119 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a2820a3-6d1e-42d0-8a89-0e47f870abc4-trusted-ca\") pod \"console-operator-7cd56b8dbc-qd9n6\" (UID: \"9a2820a3-6d1e-42d0-8a89-0e47f870abc4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:16.534780 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.534388 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f264bf41-80c4-4dda-a67e-6651b464655c-config\") pod \"service-ca-operator-544d749c8c-pfjlk\" (UID: \"f264bf41-80c4-4dda-a67e-6651b464655c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pfjlk" Nov 02 21:08:16.534874 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.534848 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-certificates\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:16.534874 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.534863 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cf65e09-2079-40cc-b597-1b67ba482a04-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:16.535460 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.535157 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08b38b87-8fd7-475e-94db-8bdd1339dc4b-ca-trust-extracted\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:16.535460 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.535352 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-stats-auth\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:16.535460 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.535436 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f264bf41-80c4-4dda-a67e-6651b464655c-serving-cert\") pod \"service-ca-operator-544d749c8c-pfjlk\" (UID: \"f264bf41-80c4-4dda-a67e-6651b464655c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pfjlk" Nov 02 21:08:16.535641 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.535618 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cf65e09-2079-40cc-b597-1b67ba482a04-tmp\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:16.535727 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:16.535708 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 21:08:16.535801 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:16.535755 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert podName:24c2ea1f-b61b-4e62-914a-2c52508bc04d nodeName:}" failed. No retries permitted until 2026-11-02 21:08:17.035741016 +0000 UTC m=+31.692815899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qpklq" (UID: "24c2ea1f-b61b-4e62-914a-2c52508bc04d") : secret "networking-console-plugin-cert" not found Nov 02 21:08:16.536408 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.536213 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7de0fc42-d787-49e7-80a2-5122a67bfba0-cert\") pod \"ingress-canary-mlgzw\" (UID: \"7de0fc42-d787-49e7-80a2-5122a67bfba0\") " pod="openshift-ingress-canary/ingress-canary-mlgzw" Nov 02 21:08:16.536408 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:16.536311 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:08:16.536408 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:16.536325 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-588654849f-w89ht: secret "image-registry-tls" not found Nov 02 21:08:16.536408 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.536340 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08b38b87-8fd7-475e-94db-8bdd1339dc4b-image-registry-private-configuration\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:16.536408 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:16.536366 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls podName:08b38b87-8fd7-475e-94db-8bdd1339dc4b nodeName:}" failed. No retries permitted until 2026-11-02 21:08:17.036350474 +0000 UTC m=+31.693425342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls") pod "image-registry-588654849f-w89ht" (UID: "08b38b87-8fd7-475e-94db-8bdd1339dc4b") : secret "image-registry-tls" not found Nov 02 21:08:16.536717 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.536527 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8cf65e09-2079-40cc-b597-1b67ba482a04-snapshots\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:16.536814 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.536791 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad2a39a-7a74-44a2-a97f-88e84e65616c-config-volume\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:16.536883 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.536853 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8cf65e09-2079-40cc-b597-1b67ba482a04-serving-cert\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:16.537287 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.537271 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-default-certificate\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:16.538236 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.538209 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08b38b87-8fd7-475e-94db-8bdd1339dc4b-installation-pull-secrets\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:16.538236 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.538229 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9a2820a3-6d1e-42d0-8a89-0e47f870abc4-serving-cert\") pod \"console-operator-7cd56b8dbc-qd9n6\" (UID: \"9a2820a3-6d1e-42d0-8a89-0e47f870abc4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:16.584272 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.584240 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf" Nov 02 21:08:16.591544 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.591468 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxcnw\" (UniqueName: \"kubernetes.io/projected/44a209b7-2144-45a0-bea0-2f8adf6658cb-kube-api-access-hxcnw\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:16.591785 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.591758 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v2wr\" (UniqueName: \"kubernetes.io/projected/8cf65e09-2079-40cc-b597-1b67ba482a04-kube-api-access-5v2wr\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:16.593895 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.593862 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-bound-sa-token\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:16.594933 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.594901 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmqww\" (UniqueName: \"kubernetes.io/projected/4ad2a39a-7a74-44a2-a97f-88e84e65616c-kube-api-access-fmqww\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:16.596997 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.596962 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxm2l\" (UniqueName: \"kubernetes.io/projected/7de0fc42-d787-49e7-80a2-5122a67bfba0-kube-api-access-jxm2l\") pod \"ingress-canary-mlgzw\" (UID: \"7de0fc42-d787-49e7-80a2-5122a67bfba0\") " pod="openshift-ingress-canary/ingress-canary-mlgzw" Nov 02 21:08:16.601987 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.601956 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w7df\" (UniqueName: \"kubernetes.io/projected/fecc158a-a288-4064-b3e9-6fe91e703f5b-kube-api-access-7w7df\") pod \"network-check-source-6f78d59f99-8jclh\" (UID: \"fecc158a-a288-4064-b3e9-6fe91e703f5b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8jclh" Nov 02 21:08:16.602254 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.602233 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxbh5\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-kube-api-access-bxbh5\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:16.609077 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.609008 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw9lp\" (UniqueName: \"kubernetes.io/projected/f264bf41-80c4-4dda-a67e-6651b464655c-kube-api-access-gw9lp\") pod \"service-ca-operator-544d749c8c-pfjlk\" (UID: \"f264bf41-80c4-4dda-a67e-6651b464655c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pfjlk" Nov 02 21:08:16.609216 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.609113 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q672k\" (UniqueName: \"kubernetes.io/projected/6b5655ce-43ca-48e4-9e4b-872283ccb85c-kube-api-access-q672k\") pod \"cluster-samples-operator-6ddb6c8b94-vmqkq\" (UID: \"6b5655ce-43ca-48e4-9e4b-872283ccb85c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" Nov 02 21:08:16.609538 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.609516 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx9qw\" (UniqueName: \"kubernetes.io/projected/934bd6cf-3dcb-40fd-9b11-97886b3f098b-kube-api-access-vx9qw\") pod \"volume-data-source-validator-58c9f5f46d-qpf24\" (UID: \"934bd6cf-3dcb-40fd-9b11-97886b3f098b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qpf24" Nov 02 21:08:16.614115 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.614081 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnvnx\" (UniqueName: \"kubernetes.io/projected/9a2820a3-6d1e-42d0-8a89-0e47f870abc4-kube-api-access-gnvnx\") pod \"console-operator-7cd56b8dbc-qd9n6\" (UID: \"9a2820a3-6d1e-42d0-8a89-0e47f870abc4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:16.619168 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.619124 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:16.644668 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.644458 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pfjlk" Nov 02 21:08:16.651421 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.651335 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:16.701332 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.675240 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8jclh" Nov 02 21:08:16.701332 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.694215 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:16.707150 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.705734 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mlgzw" Nov 02 21:08:16.900156 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.900071 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qpf24" Nov 02 21:08:16.922650 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:08:16.922623 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod345370c7_0ca8_4afe_94db_c9f5394d4d72.slice/crio-a523dc598caeeb5d5889543053a3cd0656321bcf0752eb039e61ec5e9c08e5d2 WatchSource:0}: Error finding container a523dc598caeeb5d5889543053a3cd0656321bcf0752eb039e61ec5e9c08e5d2: Status 404 returned error can't find the container with id a523dc598caeeb5d5889543053a3cd0656321bcf0752eb039e61ec5e9c08e5d2 Nov 02 21:08:16.923358 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.923338 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nv2sw"] Nov 02 21:08:16.938305 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.937066 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5zc9z"] Nov 02 21:08:16.939381 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:08:16.938557 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3acf985_6085_4061_8ca8_e6d7036cb6fd.slice/crio-43eabb098ccce10b169118e5fb17731ed32532e6e9057a0359575b38e98c8932 WatchSource:0}: Error finding container 43eabb098ccce10b169118e5fb17731ed32532e6e9057a0359575b38e98c8932: Status 404 returned error can't find the container with id 43eabb098ccce10b169118e5fb17731ed32532e6e9057a0359575b38e98c8932 Nov 02 21:08:16.948812 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.948776 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:16.948958 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.948936 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-z4kx6\" (UID: \"a899c9c5-79b4-472d-942c-5e0c4c53e954\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6" Nov 02 21:08:16.949060 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:16.949041 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 21:08:16.949240 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:16.949083 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls podName:a899c9c5-79b4-472d-942c-5e0c4c53e954 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:17.949071539 +0000 UTC m=+32.606146407 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-z4kx6" (UID: "a899c9c5-79b4-472d-942c-5e0c4c53e954") : secret "cluster-monitoring-operator-tls" not found Nov 02 21:08:16.949240 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:16.949150 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:08:16.949240 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:16.949161 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c987dc54-2p2h6: secret "image-registry-tls" not found Nov 02 21:08:16.949240 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:16.949195 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls podName:cf9df1a4-3684-464c-97f3-21a3151a7183 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:17.949184947 +0000 UTC m=+32.606259829 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls") pod "image-registry-c987dc54-2p2h6" (UID: "cf9df1a4-3684-464c-97f3-21a3151a7183") : secret "image-registry-tls" not found Nov 02 21:08:16.954358 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:08:16.952297 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a07e29_b928_4427_b4c2_5315984f03b3.slice/crio-3d12fd879df50acc9c58da47ef0db58f3bb45a29369edbe0f5eb5b196f9dec48 WatchSource:0}: Error finding container 3d12fd879df50acc9c58da47ef0db58f3bb45a29369edbe0f5eb5b196f9dec48: Status 404 returned error can't find the container with id 3d12fd879df50acc9c58da47ef0db58f3bb45a29369edbe0f5eb5b196f9dec48 Nov 02 21:08:16.954358 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.952478 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bb7st"] Nov 02 21:08:16.995529 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:16.995496 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf"] Nov 02 21:08:16.999854 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:08:16.999821 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca28f2b_c77d_4aa1_8501_ec380253ab88.slice/crio-595bbca93c4ad043d8e398da486a2883ff60f00a9aaaaf8bcec628fde4087f03 WatchSource:0}: Error finding container 595bbca93c4ad043d8e398da486a2883ff60f00a9aaaaf8bcec628fde4087f03: Status 404 returned error can't find the container with id 595bbca93c4ad043d8e398da486a2883ff60f00a9aaaaf8bcec628fde4087f03 Nov 02 21:08:17.002437 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:17.002390 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bb7st" event={"ID":"c4a07e29-b928-4427-b4c2-5315984f03b3","Type":"ContainerStarted","Data":"3d12fd879df50acc9c58da47ef0db58f3bb45a29369edbe0f5eb5b196f9dec48"} Nov 02 21:08:17.004160 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:17.004122 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5zc9z" event={"ID":"d3acf985-6085-4061-8ca8-e6d7036cb6fd","Type":"ContainerStarted","Data":"43eabb098ccce10b169118e5fb17731ed32532e6e9057a0359575b38e98c8932"} Nov 02 21:08:17.006775 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:17.006748 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-nv2sw" event={"ID":"345370c7-0ca8-4afe-94db-c9f5394d4d72","Type":"ContainerStarted","Data":"a523dc598caeeb5d5889543053a3cd0656321bcf0752eb039e61ec5e9c08e5d2"} Nov 02 21:08:17.050538 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:17.050114 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:17.050538 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:17.050181 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:17.050538 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:17.050221 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:17.050538 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:17.050255 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:08:17.050538 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:17.050277 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-588654849f-w89ht: secret "image-registry-tls" not found Nov 02 21:08:17.050538 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:17.050341 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:08:17.050538 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:17.050257 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:17.050538 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:17.050342 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls podName:08b38b87-8fd7-475e-94db-8bdd1339dc4b nodeName:}" failed. No retries permitted until 2026-11-02 21:08:18.050324197 +0000 UTC m=+32.707399089 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls") pod "image-registry-588654849f-w89ht" (UID: "08b38b87-8fd7-475e-94db-8bdd1339dc4b") : secret "image-registry-tls" not found Nov 02 21:08:17.050538 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:17.050409 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls podName:4ad2a39a-7a74-44a2-a97f-88e84e65616c nodeName:}" failed. No retries permitted until 2026-11-02 21:08:18.050394223 +0000 UTC m=+32.707469090 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls") pod "dns-default-vn62q" (UID: "4ad2a39a-7a74-44a2-a97f-88e84e65616c") : secret "dns-default-metrics-tls" not found Nov 02 21:08:17.050538 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:17.050408 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 21:08:17.050538 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:17.050442 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle podName:44a209b7-2144-45a0-bea0-2f8adf6658cb nodeName:}" failed. No retries permitted until 2026-11-02 21:08:18.050422428 +0000 UTC m=+32.707497311 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle") pod "router-default-798b7b6b45-hlcl7" (UID: "44a209b7-2144-45a0-bea0-2f8adf6658cb") : configmap references non-existent config key: service-ca.crt Nov 02 21:08:17.050538 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:17.050463 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs podName:44a209b7-2144-45a0-bea0-2f8adf6658cb nodeName:}" failed. No retries permitted until 2026-11-02 21:08:18.050454433 +0000 UTC m=+32.707529300 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs") pod "router-default-798b7b6b45-hlcl7" (UID: "44a209b7-2144-45a0-bea0-2f8adf6658cb") : secret "router-metrics-certs-default" not found Nov 02 21:08:17.050538 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:17.050528 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vmqkq\" (UID: \"6b5655ce-43ca-48e4-9e4b-872283ccb85c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" Nov 02 21:08:17.051212 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:17.050638 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qpklq\" (UID: \"24c2ea1f-b61b-4e62-914a-2c52508bc04d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qpklq" Nov 02 21:08:17.051212 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:17.050747 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 21:08:17.051212 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:17.050789 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert podName:24c2ea1f-b61b-4e62-914a-2c52508bc04d nodeName:}" failed. No retries permitted until 2026-11-02 21:08:18.050777667 +0000 UTC m=+32.707852533 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qpklq" (UID: "24c2ea1f-b61b-4e62-914a-2c52508bc04d") : secret "networking-console-plugin-cert" not found Nov 02 21:08:17.051212 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:17.050827 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 21:08:17.051212 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:17.050872 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls podName:6b5655ce-43ca-48e4-9e4b-872283ccb85c nodeName:}" failed. No retries permitted until 2026-11-02 21:08:18.05086115 +0000 UTC m=+32.707936025 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vmqkq" (UID: "6b5655ce-43ca-48e4-9e4b-872283ccb85c") : secret "samples-operator-tls" not found Nov 02 21:08:17.060173 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:17.060124 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-2ftg6"] Nov 02 21:08:17.065595 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:08:17.065563 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf65e09_2079_40cc_b597_1b67ba482a04.slice/crio-904c2e78a563c743911689439b0018c42b7c9ae7c573061d777d617010d61203 WatchSource:0}: Error finding container 904c2e78a563c743911689439b0018c42b7c9ae7c573061d777d617010d61203: Status 404 returned error can't find the container with id 904c2e78a563c743911689439b0018c42b7c9ae7c573061d777d617010d61203 Nov 02 21:08:17.079621 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:17.079161 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-pfjlk"] Nov 02 21:08:17.080050 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:08:17.080022 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf264bf41_80c4_4dda_a67e_6651b464655c.slice/crio-99213b29ed642ee2d9ddc00d9043868fee9f471355a4de7ce5e4cb2a7e78f446 WatchSource:0}: Error finding container 99213b29ed642ee2d9ddc00d9043868fee9f471355a4de7ce5e4cb2a7e78f446: Status 404 returned error can't find the container with id 99213b29ed642ee2d9ddc00d9043868fee9f471355a4de7ce5e4cb2a7e78f446 Nov 02 21:08:17.080775 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:08:17.080746 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a2820a3_6d1e_42d0_8a89_0e47f870abc4.slice/crio-26a2e1e33242c28dc251c800d13b9c6c302e9c1d7705af705aea7e307fc83ec6 WatchSource:0}: Error finding container 26a2e1e33242c28dc251c800d13b9c6c302e9c1d7705af705aea7e307fc83ec6: Status 404 returned error can't find the container with id 26a2e1e33242c28dc251c800d13b9c6c302e9c1d7705af705aea7e307fc83ec6 Nov 02 21:08:17.081041 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:17.081022 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-qd9n6"] Nov 02 21:08:17.125347 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:08:17.125321 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfecc158a_a288_4064_b3e9_6fe91e703f5b.slice/crio-fea13107742b81caf647af1279a23748237c82739f6b840f2bb100c0be1127d1 WatchSource:0}: Error finding container fea13107742b81caf647af1279a23748237c82739f6b840f2bb100c0be1127d1: Status 404 returned error can't find the container with id fea13107742b81caf647af1279a23748237c82739f6b840f2bb100c0be1127d1 Nov 02 21:08:17.127154 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:17.126967 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-8jclh"] Nov 02 21:08:17.130481 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:17.130459 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mlgzw"] Nov 02 21:08:17.355693 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:08:17.355664 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod934bd6cf_3dcb_40fd_9b11_97886b3f098b.slice/crio-a47e2648877ec7c53a35a6977dc0d961c05a330311daad6d70cfe41f3b51d198 WatchSource:0}: Error finding container a47e2648877ec7c53a35a6977dc0d961c05a330311daad6d70cfe41f3b51d198: Status 404 returned error can't find the container with id a47e2648877ec7c53a35a6977dc0d961c05a330311daad6d70cfe41f3b51d198 Nov 02 21:08:17.357007 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:17.356976 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qpf24"] Nov 02 21:08:17.766945 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:17.766915 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:08:17.767099 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:17.766955 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:08:17.767099 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:17.766922 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:08:17.822370 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:17.822338 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 21:08:17.822541 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:17.822465 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2mhzd\"" Nov 02 21:08:17.822541 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:17.822531 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wnfwz\"" Nov 02 21:08:17.822625 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:17.822619 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 21:08:17.823112 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:17.823096 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9p4p6\"" Nov 02 21:08:17.956737 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:17.956704 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-z4kx6\" (UID: \"a899c9c5-79b4-472d-942c-5e0c4c53e954\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6" Nov 02 21:08:17.957085 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:17.956774 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:17.957085 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:17.956867 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 21:08:17.957085 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:17.956908 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:08:17.957085 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:17.956921 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c987dc54-2p2h6: secret "image-registry-tls" not found Nov 02 21:08:17.957085 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:17.956949 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls podName:a899c9c5-79b4-472d-942c-5e0c4c53e954 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:19.956929568 +0000 UTC m=+34.614004462 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-z4kx6" (UID: "a899c9c5-79b4-472d-942c-5e0c4c53e954") : secret "cluster-monitoring-operator-tls" not found Nov 02 21:08:17.957085 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:17.956968 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls podName:cf9df1a4-3684-464c-97f3-21a3151a7183 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:19.956958678 +0000 UTC m=+34.614033550 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls") pod "image-registry-c987dc54-2p2h6" (UID: "cf9df1a4-3684-464c-97f3-21a3151a7183") : secret "image-registry-tls" not found Nov 02 21:08:18.012724 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:18.012271 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bb7st" event={"ID":"c4a07e29-b928-4427-b4c2-5315984f03b3","Type":"ContainerStarted","Data":"8f722c438e057b98e5979fed51090eb12a22874994fab50fda5f7de90f0d1ff2"} Nov 02 21:08:18.013902 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:18.013869 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-nv2sw" event={"ID":"345370c7-0ca8-4afe-94db-c9f5394d4d72","Type":"ContainerStarted","Data":"5c2b6d73b3cdb3a55fcb463e767fc573939cab6be001a1cd7ad4469dca22a814"} Nov 02 21:08:18.015030 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:18.015000 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8jclh" event={"ID":"fecc158a-a288-4064-b3e9-6fe91e703f5b","Type":"ContainerStarted","Data":"fea13107742b81caf647af1279a23748237c82739f6b840f2bb100c0be1127d1"} Nov 02 21:08:18.016091 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:18.016064 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" event={"ID":"9a2820a3-6d1e-42d0-8a89-0e47f870abc4","Type":"ContainerStarted","Data":"26a2e1e33242c28dc251c800d13b9c6c302e9c1d7705af705aea7e307fc83ec6"} Nov 02 21:08:18.017674 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:18.017613 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5zc9z" event={"ID":"d3acf985-6085-4061-8ca8-e6d7036cb6fd","Type":"ContainerStarted","Data":"0ba220796baed30edc63dcf5681c33b433fdfed36e329362a2565aaa071fd9e1"} Nov 02 21:08:18.018711 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:18.018690 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pfjlk" event={"ID":"f264bf41-80c4-4dda-a67e-6651b464655c","Type":"ContainerStarted","Data":"99213b29ed642ee2d9ddc00d9043868fee9f471355a4de7ce5e4cb2a7e78f446"} Nov 02 21:08:18.020008 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:18.019983 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-2ftg6" event={"ID":"8cf65e09-2079-40cc-b597-1b67ba482a04","Type":"ContainerStarted","Data":"904c2e78a563c743911689439b0018c42b7c9ae7c573061d777d617010d61203"} Nov 02 21:08:18.021171 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:18.021150 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mlgzw" event={"ID":"7de0fc42-d787-49e7-80a2-5122a67bfba0","Type":"ContainerStarted","Data":"e8c8a4724dec7748c53ba6fa336407ba878e7ff4704c694f882fcce1127403de"} Nov 02 21:08:18.022450 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:18.022409 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf" event={"ID":"5ca28f2b-c77d-4aa1-8501-ec380253ab88","Type":"ContainerStarted","Data":"595bbca93c4ad043d8e398da486a2883ff60f00a9aaaaf8bcec628fde4087f03"} Nov 02 21:08:18.023678 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:18.023659 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qpf24" event={"ID":"934bd6cf-3dcb-40fd-9b11-97886b3f098b","Type":"ContainerStarted","Data":"a47e2648877ec7c53a35a6977dc0d961c05a330311daad6d70cfe41f3b51d198"} Nov 02 21:08:18.058002 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:18.057970 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qpklq\" (UID: \"24c2ea1f-b61b-4e62-914a-2c52508bc04d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qpklq" Nov 02 21:08:18.058207 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:18.058030 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:18.058207 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:18.058055 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:18.058207 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:18.058084 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:18.058207 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:18.058116 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:18.058207 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:18.058194 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vmqkq\" (UID: \"6b5655ce-43ca-48e4-9e4b-872283ccb85c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" Nov 02 21:08:18.058446 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:18.058346 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 21:08:18.058446 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:18.058406 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls podName:6b5655ce-43ca-48e4-9e4b-872283ccb85c nodeName:}" failed. No retries permitted until 2026-11-02 21:08:20.058389627 +0000 UTC m=+34.715464520 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vmqkq" (UID: "6b5655ce-43ca-48e4-9e4b-872283ccb85c") : secret "samples-operator-tls" not found Nov 02 21:08:18.058561 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:18.058482 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 21:08:18.058561 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:18.058519 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert podName:24c2ea1f-b61b-4e62-914a-2c52508bc04d nodeName:}" failed. No retries permitted until 2026-11-02 21:08:20.058507838 +0000 UTC m=+34.715582720 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qpklq" (UID: "24c2ea1f-b61b-4e62-914a-2c52508bc04d") : secret "networking-console-plugin-cert" not found Nov 02 21:08:18.058660 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:18.058586 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:08:18.058660 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:18.058599 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-588654849f-w89ht: secret "image-registry-tls" not found Nov 02 21:08:18.058660 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:18.058631 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls podName:08b38b87-8fd7-475e-94db-8bdd1339dc4b nodeName:}" failed. No retries permitted until 2026-11-02 21:08:20.058621018 +0000 UTC m=+34.715695907 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls") pod "image-registry-588654849f-w89ht" (UID: "08b38b87-8fd7-475e-94db-8bdd1339dc4b") : secret "image-registry-tls" not found Nov 02 21:08:18.058811 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:18.058687 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 21:08:18.058811 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:18.058720 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs podName:44a209b7-2144-45a0-bea0-2f8adf6658cb nodeName:}" failed. No retries permitted until 2026-11-02 21:08:20.058709121 +0000 UTC m=+34.715784002 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs") pod "router-default-798b7b6b45-hlcl7" (UID: "44a209b7-2144-45a0-bea0-2f8adf6658cb") : secret "router-metrics-certs-default" not found Nov 02 21:08:18.058811 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:18.058783 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle podName:44a209b7-2144-45a0-bea0-2f8adf6658cb nodeName:}" failed. No retries permitted until 2026-11-02 21:08:20.058772614 +0000 UTC m=+34.715847496 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle") pod "router-default-798b7b6b45-hlcl7" (UID: "44a209b7-2144-45a0-bea0-2f8adf6658cb") : configmap references non-existent config key: service-ca.crt Nov 02 21:08:18.058938 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:18.058838 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:08:18.058938 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:18.058867 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls podName:4ad2a39a-7a74-44a2-a97f-88e84e65616c nodeName:}" failed. No retries permitted until 2026-11-02 21:08:20.058858235 +0000 UTC m=+34.715933107 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls") pod "dns-default-vn62q" (UID: "4ad2a39a-7a74-44a2-a97f-88e84e65616c") : secret "dns-default-metrics-tls" not found Nov 02 21:08:18.273291 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:18.273196 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-bb7st" podStartSLOduration=385.273182236 podStartE2EDuration="6m25.273182236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:18.271847194 +0000 UTC m=+32.928922095" watchObservedRunningTime="2026-11-02 21:08:18.273182236 +0000 UTC m=+32.930257164" Nov 02 21:08:18.619587 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:18.619480 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-nv2sw" podStartSLOduration=385.619461316 podStartE2EDuration="6m25.619461316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:18.618712459 +0000 UTC m=+33.275787340" watchObservedRunningTime="2026-11-02 21:08:18.619461316 +0000 UTC m=+33.276536206" Nov 02 21:08:18.621098 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:18.621051 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-5zc9z" podStartSLOduration=385.621035357 podStartE2EDuration="6m25.621035357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:18.504451029 +0000 UTC m=+33.161525920" watchObservedRunningTime="2026-11-02 21:08:18.621035357 +0000 UTC m=+33.278110246" Nov 02 21:08:18.767940 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:18.766840 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-original-pull-secret\") pod \"global-pull-secret-syncer-bllxh\" (UID: \"87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1\") " pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:08:18.767940 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:18.766960 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs\") pod \"network-metrics-daemon-s98xl\" (UID: \"d45d9164-2ba4-400d-8b27-273d7f4f351d\") " pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:08:18.767940 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:18.767183 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 21:08:18.767940 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:18.767249 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs podName:d45d9164-2ba4-400d-8b27-273d7f4f351d nodeName:}" failed. No retries permitted until 2026-11-02 21:08:50.767232209 +0000 UTC m=+65.424307091 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs") pod "network-metrics-daemon-s98xl" (UID: "d45d9164-2ba4-400d-8b27-273d7f4f351d") : secret "metrics-daemon-secret" not found Nov 02 21:08:18.777581 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:18.777548 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-original-pull-secret\") pod \"global-pull-secret-syncer-bllxh\" (UID: \"87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1\") " pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:08:18.868939 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:18.868337 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfbgk\" (UniqueName: \"kubernetes.io/projected/c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7-kube-api-access-lfbgk\") pod \"network-check-target-8c67p\" (UID: \"c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7\") " pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:08:18.871747 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:18.871657 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfbgk\" (UniqueName: \"kubernetes.io/projected/c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7-kube-api-access-lfbgk\") pod \"network-check-target-8c67p\" (UID: \"c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7\") " pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:08:19.008091 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:19.007871 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:08:19.031449 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:19.031062 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:08:19.428285 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:19.426808 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bllxh"] Nov 02 21:08:19.428371 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:08:19.428284 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87565769_a30f_4ce5_b0d4_0bb6ccf2a4f1.slice/crio-d5d4f6ec3bb69138dd6f297adbe5cc65f427c366c7e7e872caa1ea7bd8e267af WatchSource:0}: Error finding container d5d4f6ec3bb69138dd6f297adbe5cc65f427c366c7e7e872caa1ea7bd8e267af: Status 404 returned error can't find the container with id d5d4f6ec3bb69138dd6f297adbe5cc65f427c366c7e7e872caa1ea7bd8e267af Nov 02 21:08:19.469492 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:19.469463 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8c67p"] Nov 02 21:08:19.469862 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:08:19.469832 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc96ef2e1_52b0_4f0d_8e28_0d1dfa8fdfb7.slice/crio-996e3208f62bddecd270a96f63112c517e921203b4ff8047fa02bb22f44c06d1 WatchSource:0}: Error finding container 996e3208f62bddecd270a96f63112c517e921203b4ff8047fa02bb22f44c06d1: Status 404 returned error can't find the container with id 996e3208f62bddecd270a96f63112c517e921203b4ff8047fa02bb22f44c06d1 Nov 02 21:08:19.708829 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:19.708737 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:19.910570 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:19.910527 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:19.981622 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:19.980822 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-z4kx6\" (UID: \"a899c9c5-79b4-472d-942c-5e0c4c53e954\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6" Nov 02 21:08:19.981622 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:19.980916 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:19.981622 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:19.981069 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:08:19.981622 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:19.981085 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c987dc54-2p2h6: secret "image-registry-tls" not found Nov 02 21:08:19.981622 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:19.981162 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls podName:cf9df1a4-3684-464c-97f3-21a3151a7183 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:23.981143733 +0000 UTC m=+38.638218617 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls") pod "image-registry-c987dc54-2p2h6" (UID: "cf9df1a4-3684-464c-97f3-21a3151a7183") : secret "image-registry-tls" not found Nov 02 21:08:19.981622 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:19.981511 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 21:08:19.981622 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:19.981548 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls podName:a899c9c5-79b4-472d-942c-5e0c4c53e954 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:23.981537482 +0000 UTC m=+38.638612349 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-z4kx6" (UID: "a899c9c5-79b4-472d-942c-5e0c4c53e954") : secret "cluster-monitoring-operator-tls" not found Nov 02 21:08:20.032781 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:20.032721 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8c67p" event={"ID":"c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7","Type":"ContainerStarted","Data":"996e3208f62bddecd270a96f63112c517e921203b4ff8047fa02bb22f44c06d1"} Nov 02 21:08:20.035009 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:20.034943 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bllxh" event={"ID":"87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1","Type":"ContainerStarted","Data":"d5d4f6ec3bb69138dd6f297adbe5cc65f427c366c7e7e872caa1ea7bd8e267af"} Nov 02 21:08:20.083163 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:20.082411 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:20.083163 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:20.082465 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:20.083163 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:20.082510 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vmqkq\" (UID: \"6b5655ce-43ca-48e4-9e4b-872283ccb85c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" Nov 02 21:08:20.083163 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:20.082563 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qpklq\" (UID: \"24c2ea1f-b61b-4e62-914a-2c52508bc04d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qpklq" Nov 02 21:08:20.083163 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:20.082608 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:20.083163 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:20.082627 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:20.083163 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:20.082732 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 21:08:20.083163 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:20.082784 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs podName:44a209b7-2144-45a0-bea0-2f8adf6658cb nodeName:}" failed. No retries permitted until 2026-11-02 21:08:24.082770671 +0000 UTC m=+38.739845552 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs") pod "router-default-798b7b6b45-hlcl7" (UID: "44a209b7-2144-45a0-bea0-2f8adf6658cb") : secret "router-metrics-certs-default" not found Nov 02 21:08:20.083163 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:20.082839 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle podName:44a209b7-2144-45a0-bea0-2f8adf6658cb nodeName:}" failed. No retries permitted until 2026-11-02 21:08:24.082832162 +0000 UTC m=+38.739907028 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle") pod "router-default-798b7b6b45-hlcl7" (UID: "44a209b7-2144-45a0-bea0-2f8adf6658cb") : configmap references non-existent config key: service-ca.crt Nov 02 21:08:20.083163 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:20.082878 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:08:20.083163 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:20.082904 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls podName:4ad2a39a-7a74-44a2-a97f-88e84e65616c nodeName:}" failed. No retries permitted until 2026-11-02 21:08:24.082894742 +0000 UTC m=+38.739969623 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls") pod "dns-default-vn62q" (UID: "4ad2a39a-7a74-44a2-a97f-88e84e65616c") : secret "dns-default-metrics-tls" not found Nov 02 21:08:20.083163 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:20.082956 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 21:08:20.083163 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:20.082975 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls podName:6b5655ce-43ca-48e4-9e4b-872283ccb85c nodeName:}" failed. No retries permitted until 2026-11-02 21:08:24.082969828 +0000 UTC m=+38.740044695 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vmqkq" (UID: "6b5655ce-43ca-48e4-9e4b-872283ccb85c") : secret "samples-operator-tls" not found Nov 02 21:08:20.083163 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:20.083008 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 21:08:20.083163 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:20.083031 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert podName:24c2ea1f-b61b-4e62-914a-2c52508bc04d nodeName:}" failed. No retries permitted until 2026-11-02 21:08:24.083023199 +0000 UTC m=+38.740098066 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qpklq" (UID: "24c2ea1f-b61b-4e62-914a-2c52508bc04d") : secret "networking-console-plugin-cert" not found Nov 02 21:08:20.083984 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:20.083075 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:08:20.083984 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:20.083083 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-588654849f-w89ht: secret "image-registry-tls" not found Nov 02 21:08:20.083984 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:20.083102 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls podName:08b38b87-8fd7-475e-94db-8bdd1339dc4b nodeName:}" failed. No retries permitted until 2026-11-02 21:08:24.083095346 +0000 UTC m=+38.740170226 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls") pod "image-registry-588654849f-w89ht" (UID: "08b38b87-8fd7-475e-94db-8bdd1339dc4b") : secret "image-registry-tls" not found Nov 02 21:08:20.628210 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:20.628111 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:21.039954 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:21.039921 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/0.log" Nov 02 21:08:21.040426 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:21.039964 2577 generic.go:356] "Generic (PLEG): container finished" podID="9a2820a3-6d1e-42d0-8a89-0e47f870abc4" containerID="63c7254adbd76619e67740b0c083548ac63f800cc1cb23ebc3397b02d8a3d3c9" exitCode=255 Nov 02 21:08:21.040426 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:21.040069 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" event={"ID":"9a2820a3-6d1e-42d0-8a89-0e47f870abc4","Type":"ContainerDied","Data":"63c7254adbd76619e67740b0c083548ac63f800cc1cb23ebc3397b02d8a3d3c9"} Nov 02 21:08:21.040540 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:21.040461 2577 scope.go:122] "RemoveContainer" containerID="63c7254adbd76619e67740b0c083548ac63f800cc1cb23ebc3397b02d8a3d3c9" Nov 02 21:08:21.044654 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:21.043821 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qpf24" event={"ID":"934bd6cf-3dcb-40fd-9b11-97886b3f098b","Type":"ContainerStarted","Data":"de25c58931d42528d2700dfedfcbeeed4f9f117697882f899e46989027bdf927"} Nov 02 21:08:21.443223 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:21.443076 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:21.707053 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:21.706974 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:21.901833 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:21.901255 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:22.051645 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:22.051616 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/1.log" Nov 02 21:08:22.053906 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:22.053887 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/0.log" Nov 02 21:08:22.054035 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:22.053924 2577 generic.go:356] "Generic (PLEG): container finished" podID="9a2820a3-6d1e-42d0-8a89-0e47f870abc4" containerID="25880bea858c09ebae11d9d77d74a3d9d1f8c364288f4a9ac3048abab9b30755" exitCode=255 Nov 02 21:08:22.054035 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:22.054018 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" event={"ID":"9a2820a3-6d1e-42d0-8a89-0e47f870abc4","Type":"ContainerDied","Data":"25880bea858c09ebae11d9d77d74a3d9d1f8c364288f4a9ac3048abab9b30755"} Nov 02 21:08:22.054152 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:22.054072 2577 scope.go:122] "RemoveContainer" containerID="63c7254adbd76619e67740b0c083548ac63f800cc1cb23ebc3397b02d8a3d3c9" Nov 02 21:08:22.054474 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:22.054460 2577 scope.go:122] "RemoveContainer" containerID="25880bea858c09ebae11d9d77d74a3d9d1f8c364288f4a9ac3048abab9b30755" Nov 02 21:08:22.054649 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:22.054631 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qd9n6_openshift-console-operator(9a2820a3-6d1e-42d0-8a89-0e47f870abc4)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" podUID="9a2820a3-6d1e-42d0-8a89-0e47f870abc4" Nov 02 21:08:22.060028 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:22.060001 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-2ftg6" event={"ID":"8cf65e09-2079-40cc-b597-1b67ba482a04","Type":"ContainerStarted","Data":"d4b1da20e3daf85a69892deda5cd65868da955b9044216326d8e6013f8a91a44"} Nov 02 21:08:22.180031 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:22.179967 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qpf24" podStartSLOduration=353.450622869 podStartE2EDuration="5m56.179951488s" totalImagesPullingTime="2.729328619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:02:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:21.274059313 +0000 UTC m=+35.931134227" watchObservedRunningTime="2026-11-02 21:08:22.179951488 +0000 UTC m=+36.837026435" Nov 02 21:08:22.297306 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:22.297253 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-2ftg6" podStartSLOduration=351.585604026 podStartE2EDuration="5m56.297237727s" totalImagesPullingTime="4.711633701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:02:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:22.296227999 +0000 UTC m=+36.953302905" watchObservedRunningTime="2026-11-02 21:08:22.297237727 +0000 UTC m=+36.954312618" Nov 02 21:08:23.067406 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:23.067254 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/1.log" Nov 02 21:08:23.067818 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:23.067691 2577 scope.go:122] "RemoveContainer" containerID="25880bea858c09ebae11d9d77d74a3d9d1f8c364288f4a9ac3048abab9b30755" Nov 02 21:08:23.067967 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:23.067898 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qd9n6_openshift-console-operator(9a2820a3-6d1e-42d0-8a89-0e47f870abc4)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" podUID="9a2820a3-6d1e-42d0-8a89-0e47f870abc4" Nov 02 21:08:23.285610 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:23.285573 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2nshl"] Nov 02 21:08:23.304314 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:23.304287 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:23.329815 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:23.329781 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2nshl"] Nov 02 21:08:23.358314 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:23.358287 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 21:08:23.358454 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:23.358416 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9hv44\"" Nov 02 21:08:23.358525 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:23.358510 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 21:08:23.417917 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:23.417885 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-crio-socket\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:23.418088 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:23.417951 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:23.418088 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:23.418016 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:23.418223 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:23.418200 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx8dl\" (UniqueName: \"kubernetes.io/projected/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-kube-api-access-jx8dl\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:23.418358 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:23.418340 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-data-volume\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:23.519513 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:23.519477 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-data-volume\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:23.519654 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:23.519552 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-crio-socket\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:23.519654 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:23.519601 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:23.519654 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:23.519626 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:23.519826 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:23.519695 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx8dl\" (UniqueName: \"kubernetes.io/projected/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-kube-api-access-jx8dl\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:23.520378 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:23.520355 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 21:08:23.520476 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:23.520414 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-kube-rbac-proxy-cm podName:67c3caa0-0b53-42c9-98b2-96addd3dd7fc nodeName:}" failed. No retries permitted until 2026-11-02 21:08:24.02039855 +0000 UTC m=+38.677473418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2nshl" (UID: "67c3caa0-0b53-42c9-98b2-96addd3dd7fc") : configmap "kube-rbac-proxy" not found Nov 02 21:08:23.520732 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:23.520535 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-crio-socket\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:23.520732 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:23.520648 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:08:23.520732 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:23.520707 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls podName:67c3caa0-0b53-42c9-98b2-96addd3dd7fc nodeName:}" failed. No retries permitted until 2026-11-02 21:08:24.020692634 +0000 UTC m=+38.677767521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2nshl" (UID: "67c3caa0-0b53-42c9-98b2-96addd3dd7fc") : secret "insights-runtime-extractor-tls" not found Nov 02 21:08:23.520854 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:23.520838 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-data-volume\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:23.579105 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:23.579074 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx8dl\" (UniqueName: \"kubernetes.io/projected/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-kube-api-access-jx8dl\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:23.584190 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:23.584125 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:23.791232 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:23.791197 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:23.992418 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:23.992385 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:24.025593 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.025557 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-z4kx6\" (UID: \"a899c9c5-79b4-472d-942c-5e0c4c53e954\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6" Nov 02 21:08:24.025734 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.025628 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:24.025734 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.025669 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:24.025734 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.025685 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:24.025851 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:24.025837 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:08:24.025899 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:24.025889 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls podName:67c3caa0-0b53-42c9-98b2-96addd3dd7fc nodeName:}" failed. No retries permitted until 2026-11-02 21:08:25.025877391 +0000 UTC m=+39.682952259 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2nshl" (UID: "67c3caa0-0b53-42c9-98b2-96addd3dd7fc") : secret "insights-runtime-extractor-tls" not found Nov 02 21:08:24.025951 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:24.025941 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 21:08:24.025982 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:24.025965 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls podName:a899c9c5-79b4-472d-942c-5e0c4c53e954 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:32.025957873 +0000 UTC m=+46.683032739 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-z4kx6" (UID: "a899c9c5-79b4-472d-942c-5e0c4c53e954") : secret "cluster-monitoring-operator-tls" not found Nov 02 21:08:24.026025 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:24.026014 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:08:24.026025 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:24.026025 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c987dc54-2p2h6: secret "image-registry-tls" not found Nov 02 21:08:24.026087 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:24.026048 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls podName:cf9df1a4-3684-464c-97f3-21a3151a7183 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:32.026039 +0000 UTC m=+46.683113868 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls") pod "image-registry-c987dc54-2p2h6" (UID: "cf9df1a4-3684-464c-97f3-21a3151a7183") : secret "image-registry-tls" not found Nov 02 21:08:24.026087 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:24.026085 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 21:08:24.026200 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:24.026106 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-kube-rbac-proxy-cm podName:67c3caa0-0b53-42c9-98b2-96addd3dd7fc nodeName:}" failed. No retries permitted until 2026-11-02 21:08:25.026100545 +0000 UTC m=+39.683175412 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2nshl" (UID: "67c3caa0-0b53-42c9-98b2-96addd3dd7fc") : configmap "kube-rbac-proxy" not found Nov 02 21:08:24.077507 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.077419 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf" event={"ID":"5ca28f2b-c77d-4aa1-8501-ec380253ab88","Type":"ContainerStarted","Data":"c0adfd7d26ac62535fc6317ff0ddfbc014f0677a42b88aeba041e2fd2b04a037"} Nov 02 21:08:24.079835 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.079799 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8c67p" event={"ID":"c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7","Type":"ContainerStarted","Data":"6158d47c3ffefa2b57fd77785680eaf0956bcc802f3c6c5fcb3d5ce5138ee841"} Nov 02 21:08:24.080350 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.080331 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:08:24.082430 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.082399 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8jclh" event={"ID":"fecc158a-a288-4064-b3e9-6fe91e703f5b","Type":"ContainerStarted","Data":"1f5e1e62d903b4338ae81db4c714a54a19ae833cfae1a2884a71cfad7ea9ce36"} Nov 02 21:08:24.084224 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.084202 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pfjlk" event={"ID":"f264bf41-80c4-4dda-a67e-6651b464655c","Type":"ContainerStarted","Data":"6dd665d446837b1ec657406834feb0025c54d6a33b5795993c88ea56dd5b1835"} Nov 02 21:08:24.127466 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.126510 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:24.127466 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.126556 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:24.127466 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.126590 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:24.127466 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.126621 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:24.127466 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.126675 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vmqkq\" (UID: \"6b5655ce-43ca-48e4-9e4b-872283ccb85c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" Nov 02 21:08:24.127466 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.126758 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qpklq\" (UID: \"24c2ea1f-b61b-4e62-914a-2c52508bc04d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qpklq" Nov 02 21:08:24.127466 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:24.126921 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 21:08:24.127466 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:24.126997 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert podName:24c2ea1f-b61b-4e62-914a-2c52508bc04d nodeName:}" failed. No retries permitted until 2026-11-02 21:08:32.126978767 +0000 UTC m=+46.784053654 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qpklq" (UID: "24c2ea1f-b61b-4e62-914a-2c52508bc04d") : secret "networking-console-plugin-cert" not found Nov 02 21:08:24.127466 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:24.127066 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:08:24.127466 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:24.127076 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-588654849f-w89ht: secret "image-registry-tls" not found Nov 02 21:08:24.127466 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:24.127107 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls podName:08b38b87-8fd7-475e-94db-8bdd1339dc4b nodeName:}" failed. No retries permitted until 2026-11-02 21:08:32.127096576 +0000 UTC m=+46.784171457 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls") pod "image-registry-588654849f-w89ht" (UID: "08b38b87-8fd7-475e-94db-8bdd1339dc4b") : secret "image-registry-tls" not found Nov 02 21:08:24.127466 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:24.127181 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 21:08:24.127466 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:24.127213 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs podName:44a209b7-2144-45a0-bea0-2f8adf6658cb nodeName:}" failed. No retries permitted until 2026-11-02 21:08:32.127202897 +0000 UTC m=+46.784277778 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs") pod "router-default-798b7b6b45-hlcl7" (UID: "44a209b7-2144-45a0-bea0-2f8adf6658cb") : secret "router-metrics-certs-default" not found Nov 02 21:08:24.127466 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:24.127273 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle podName:44a209b7-2144-45a0-bea0-2f8adf6658cb nodeName:}" failed. No retries permitted until 2026-11-02 21:08:32.127264279 +0000 UTC m=+46.784339162 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle") pod "router-default-798b7b6b45-hlcl7" (UID: "44a209b7-2144-45a0-bea0-2f8adf6658cb") : configmap references non-existent config key: service-ca.crt Nov 02 21:08:24.127466 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:24.127322 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:08:24.128819 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:24.127352 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls podName:4ad2a39a-7a74-44a2-a97f-88e84e65616c nodeName:}" failed. No retries permitted until 2026-11-02 21:08:32.127342636 +0000 UTC m=+46.784417518 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls") pod "dns-default-vn62q" (UID: "4ad2a39a-7a74-44a2-a97f-88e84e65616c") : secret "dns-default-metrics-tls" not found Nov 02 21:08:24.128819 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:24.127403 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 21:08:24.128819 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:24.127432 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls podName:6b5655ce-43ca-48e4-9e4b-872283ccb85c nodeName:}" failed. No retries permitted until 2026-11-02 21:08:32.127421923 +0000 UTC m=+46.784496805 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vmqkq" (UID: "6b5655ce-43ca-48e4-9e4b-872283ccb85c") : secret "samples-operator-tls" not found Nov 02 21:08:24.194612 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.194560 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf" podStartSLOduration=352.227973478 podStartE2EDuration="5m58.19454405s" totalImagesPullingTime="5.966570572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:02:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:24.193463539 +0000 UTC m=+38.850538453" watchObservedRunningTime="2026-11-02 21:08:24.19454405 +0000 UTC m=+38.851618939" Nov 02 21:08:24.311158 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.310341 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8jclh" podStartSLOduration=349.729048274 podStartE2EDuration="5m56.310320753s" totalImagesPullingTime="6.581272479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:02:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:24.309639028 +0000 UTC m=+38.966713910" watchObservedRunningTime="2026-11-02 21:08:24.310320753 +0000 UTC m=+38.967395643" Nov 02 21:08:24.426523 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.426406 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8c67p" podStartSLOduration=34.028333127 podStartE2EDuration="38.426388764s" totalImagesPullingTime="4.398055637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:24.425155561 +0000 UTC m=+39.082230468" watchObservedRunningTime="2026-11-02 21:08:24.426388764 +0000 UTC m=+39.083463655" Nov 02 21:08:24.544381 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.544320 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pfjlk" podStartSLOduration=352.167888939 podStartE2EDuration="5m58.544302926s" totalImagesPullingTime="6.376413987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:02:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:24.543030898 +0000 UTC m=+39.200106010" watchObservedRunningTime="2026-11-02 21:08:24.544302926 +0000 UTC m=+39.201377839" Nov 02 21:08:24.654263 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.654192 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:24.778731 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.778698 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-wf84d"] Nov 02 21:08:24.790446 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.790408 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wf84d" Nov 02 21:08:24.833717 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.833678 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-wf84d"] Nov 02 21:08:24.833993 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.833966 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kmdm\" (UniqueName: \"kubernetes.io/projected/39830cb0-c259-446d-b2d3-fd8956b968fb-kube-api-access-9kmdm\") pod \"migrator-66df5d496b-wf84d\" (UID: \"39830cb0-c259-446d-b2d3-fd8956b968fb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wf84d" Nov 02 21:08:24.845507 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.845479 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 21:08:24.845635 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.845626 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jb7zz\"" Nov 02 21:08:24.845719 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.845706 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 21:08:24.864166 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.864108 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:24.935447 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.935393 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kmdm\" (UniqueName: \"kubernetes.io/projected/39830cb0-c259-446d-b2d3-fd8956b968fb-kube-api-access-9kmdm\") pod \"migrator-66df5d496b-wf84d\" (UID: \"39830cb0-c259-446d-b2d3-fd8956b968fb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wf84d" Nov 02 21:08:24.993614 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:24.993582 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kmdm\" (UniqueName: \"kubernetes.io/projected/39830cb0-c259-446d-b2d3-fd8956b968fb-kube-api-access-9kmdm\") pod \"migrator-66df5d496b-wf84d\" (UID: \"39830cb0-c259-446d-b2d3-fd8956b968fb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wf84d" Nov 02 21:08:25.036976 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:25.036899 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:25.036976 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:25.036931 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:25.037168 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:25.037016 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 21:08:25.037168 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:25.037030 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:08:25.037168 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:25.037090 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-kube-rbac-proxy-cm podName:67c3caa0-0b53-42c9-98b2-96addd3dd7fc nodeName:}" failed. No retries permitted until 2026-11-02 21:08:27.037073741 +0000 UTC m=+41.694148631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2nshl" (UID: "67c3caa0-0b53-42c9-98b2-96addd3dd7fc") : configmap "kube-rbac-proxy" not found Nov 02 21:08:25.037270 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:25.037220 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls podName:67c3caa0-0b53-42c9-98b2-96addd3dd7fc nodeName:}" failed. No retries permitted until 2026-11-02 21:08:27.037207766 +0000 UTC m=+41.694282636 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2nshl" (UID: "67c3caa0-0b53-42c9-98b2-96addd3dd7fc") : secret "insights-runtime-extractor-tls" not found Nov 02 21:08:25.074776 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:25.074745 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:25.113320 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:25.113291 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wf84d" Nov 02 21:08:25.524155 ip-10-0-128-89 kubenswrapper[2577]: W1102 21:08:25.520362 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39830cb0_c259_446d_b2d3_fd8956b968fb.slice/crio-24cf1e198be9f0535676921dffad7301a8eec044161ab4b7507aa1fe58f1dc6d WatchSource:0}: Error finding container 24cf1e198be9f0535676921dffad7301a8eec044161ab4b7507aa1fe58f1dc6d: Status 404 returned error can't find the container with id 24cf1e198be9f0535676921dffad7301a8eec044161ab4b7507aa1fe58f1dc6d Nov 02 21:08:25.524155 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:25.522261 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-wf84d"] Nov 02 21:08:25.783583 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:25.783502 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:25.983206 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:25.983168 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:26.093070 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:26.092972 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mlgzw" event={"ID":"7de0fc42-d787-49e7-80a2-5122a67bfba0","Type":"ContainerStarted","Data":"f7c057fa028d9879d9eaa679947238cf0a733d2c2a6bd7eefe6319f23822a709"} Nov 02 21:08:26.094438 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:26.094397 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wf84d" event={"ID":"39830cb0-c259-446d-b2d3-fd8956b968fb","Type":"ContainerStarted","Data":"24cf1e198be9f0535676921dffad7301a8eec044161ab4b7507aa1fe58f1dc6d"} Nov 02 21:08:26.186847 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:26.186808 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:26.210318 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:26.210258 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mlgzw" podStartSLOduration=2.253924964 podStartE2EDuration="10.210241209s" totalImagesPullingTime="7.956316245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:08:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:08:26.208635467 +0000 UTC m=+40.865710406" watchObservedRunningTime="2026-11-02 21:08:26.210241209 +0000 UTC m=+40.867316104" Nov 02 21:08:26.652572 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:26.652534 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:26.652765 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:26.652601 2577 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:26.653071 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:26.653047 2577 scope.go:122] "RemoveContainer" containerID="25880bea858c09ebae11d9d77d74a3d9d1f8c364288f4a9ac3048abab9b30755" Nov 02 21:08:26.653320 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:26.653294 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qd9n6_openshift-console-operator(9a2820a3-6d1e-42d0-8a89-0e47f870abc4)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" podUID="9a2820a3-6d1e-42d0-8a89-0e47f870abc4" Nov 02 21:08:27.058074 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:27.058038 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:27.058074 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:27.058077 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:27.058337 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:27.058202 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 21:08:27.058337 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:27.058221 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:08:27.058337 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:27.058285 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-kube-rbac-proxy-cm podName:67c3caa0-0b53-42c9-98b2-96addd3dd7fc nodeName:}" failed. No retries permitted until 2026-11-02 21:08:31.05826519 +0000 UTC m=+45.715340080 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2nshl" (UID: "67c3caa0-0b53-42c9-98b2-96addd3dd7fc") : configmap "kube-rbac-proxy" not found Nov 02 21:08:27.058337 ip-10-0-128-89 kubenswrapper[2577]: E1102 21:08:27.058308 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls podName:67c3caa0-0b53-42c9-98b2-96addd3dd7fc nodeName:}" failed. No retries permitted until 2026-11-02 21:08:31.058298707 +0000 UTC m=+45.715373577 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2nshl" (UID: "67c3caa0-0b53-42c9-98b2-96addd3dd7fc") : secret "insights-runtime-extractor-tls" not found Nov 02 21:08:27.210208 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:27.210110 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:27.442143 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:27.442045 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:27.471170 ip-10-0-128-89 kubenswrapper[2577]: I1102 21:08:27.471066 2577 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:08:27.471256 ip-10-0-128-89 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 21:08:27.490531 ip-10-0-128-89 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 21:08:27.490701 ip-10-0-128-89 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 21:08:27.490751 ip-10-0-128-89 systemd[1]: kubelet.service: Consumed 3.127s CPU time. Nov 02 21:08:27.501728 ip-10-0-128-89 systemd[1]: Starting Kubernetes Kubelet... Nov 02 21:08:27.590462 ip-10-0-128-89 kubenswrapper[4935]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:08:27.590462 ip-10-0-128-89 kubenswrapper[4935]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 21:08:27.590462 ip-10-0-128-89 kubenswrapper[4935]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:08:27.590462 ip-10-0-128-89 kubenswrapper[4935]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:08:27.592799 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592706 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:08:27.592799 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592798 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:08:27.592799 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592803 4935 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:08:27.592897 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592806 4935 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:08:27.592897 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592810 4935 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:08:27.592897 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592813 4935 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:08:27.592897 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592816 4935 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:08:27.592897 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592819 4935 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:08:27.592897 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592822 4935 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:08:27.592897 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592825 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:08:27.592897 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592827 4935 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:08:27.592897 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592830 4935 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:08:27.592897 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592832 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:08:27.592897 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592835 4935 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:08:27.592897 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592838 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:08:27.592897 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592841 4935 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:08:27.592897 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592846 4935 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:08:27.592897 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592850 4935 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:08:27.592897 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592855 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:08:27.592897 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592859 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:08:27.592897 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592862 4935 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:08:27.592897 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592865 4935 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:08:27.592897 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592868 4935 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:08:27.593412 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592877 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:08:27.593412 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592880 4935 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:08:27.593412 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592883 4935 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:08:27.593412 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592886 4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:08:27.593412 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592889 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:08:27.593412 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592891 4935 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:08:27.593412 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592895 4935 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:08:27.593412 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592898 4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:08:27.593412 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592900 4935 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:08:27.593412 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592903 4935 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:08:27.593412 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592906 4935 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:08:27.593412 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592909 4935 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:08:27.593412 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592913 4935 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:08:27.593412 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592916 4935 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:08:27.593412 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592918 4935 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:08:27.593412 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592922 4935 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:08:27.593412 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592926 4935 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:08:27.593412 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592933 4935 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:08:27.593412 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592939 4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:08:27.593890 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592943 4935 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:08:27.593890 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592948 4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:08:27.593890 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592950 4935 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:08:27.593890 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592954 4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:08:27.593890 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592957 4935 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:08:27.593890 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592960 4935 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:08:27.593890 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592963 4935 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:08:27.593890 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592966 4935 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:08:27.593890 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592968 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:08:27.593890 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592971 4935 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:08:27.593890 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592974 4935 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:08:27.593890 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592977 4935 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:08:27.593890 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592980 4935 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:08:27.593890 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592982 4935 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:08:27.593890 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592985 4935 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:08:27.593890 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592988 4935 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:08:27.593890 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592991 4935 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:08:27.593890 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592994 4935 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:08:27.593890 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592996 4935 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:08:27.593890 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.592999 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:08:27.594436 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593004 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:08:27.594436 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593008 4935 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:08:27.594436 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593013 4935 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:08:27.594436 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593017 4935 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:08:27.594436 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593020 4935 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:08:27.594436 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593023 4935 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:08:27.594436 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593026 4935 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:08:27.594436 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593029 4935 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:08:27.594436 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593031 4935 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:08:27.594436 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593034 4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:08:27.594436 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593037 4935 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:08:27.594436 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593039 4935 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:08:27.594436 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593042 4935 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:08:27.594436 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593044 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:08:27.594436 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593048 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:08:27.594436 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593051 4935 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:08:27.594436 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593053 4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:08:27.594436 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593056 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:08:27.594436 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593059 4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:08:27.594971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593061 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:08:27.594971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593064 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:08:27.594971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593068 4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:08:27.594971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593071 4935 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:08:27.594971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593073 4935 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:08:27.594971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593077 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:08:27.594971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593080 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:08:27.594971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593084 4935 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:08:27.594971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593089 4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:08:27.594971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593093 4935 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:08:27.594971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593096 4935 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:08:27.594971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593099 4935 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:08:27.594971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593102 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:08:27.594971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593104 4935 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:08:27.594971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593107 4935 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:08:27.594971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593110 4935 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:08:27.594971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593112 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:08:27.594971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593115 4935 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:08:27.594971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593117 4935 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:08:27.595467 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593120 4935 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:08:27.595467 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593122 4935 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:08:27.595467 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593124 4935 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:08:27.595467 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593180 4935 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:08:27.595467 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593184 4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:08:27.595467 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.593189 4935 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:08:27.595467 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594013 4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:08:27.595467 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594021 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:08:27.595467 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594025 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:08:27.595467 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594029 4935 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:08:27.595467 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594032 4935 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:08:27.595467 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594035 4935 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:08:27.595467 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594039 4935 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:08:27.595467 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594041 4935 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:08:27.595467 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594044 4935 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:08:27.595467 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594047 4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:08:27.595467 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594050 4935 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:08:27.595467 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594053 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:08:27.595467 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594056 4935 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:08:27.595968 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594058 4935 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:08:27.595968 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594061 4935 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:08:27.595968 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594063 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:08:27.595968 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594066 4935 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:08:27.595968 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594069 4935 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:08:27.595968 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594071 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:08:27.595968 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594074 4935 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:08:27.595968 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594077 4935 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:08:27.595968 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594080 4935 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:08:27.595968 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594082 4935 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:08:27.595968 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594086 4935 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:08:27.595968 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594090 4935 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:08:27.595968 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594094 4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:08:27.595968 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594099 4935 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:08:27.595968 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594103 4935 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:08:27.595968 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594106 4935 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:08:27.595968 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594109 4935 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:08:27.595968 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594111 4935 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:08:27.595968 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594114 4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:08:27.595968 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594118 4935 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:08:27.596486 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594120 4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:08:27.596486 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594123 4935 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:08:27.596486 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594126 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:08:27.596486 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594141 4935 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:08:27.596486 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594144 4935 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:08:27.596486 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594147 4935 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:08:27.596486 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594150 4935 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:08:27.596486 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594153 4935 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:08:27.596486 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594157 4935 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:08:27.596486 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594162 4935 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:08:27.596486 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594165 4935 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:08:27.596486 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594169 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:08:27.596486 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594172 4935 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:08:27.596486 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594175 4935 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:08:27.596486 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594178 4935 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:08:27.596486 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594180 4935 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:08:27.596486 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594183 4935 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:08:27.596486 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594185 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:08:27.596486 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594188 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:08:27.596970 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594191 4935 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:08:27.596970 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594193 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:08:27.596970 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594196 4935 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:08:27.596970 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594199 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:08:27.596970 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594201 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:08:27.596970 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594204 4935 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:08:27.596970 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594206 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:08:27.596970 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594209 4935 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:08:27.596970 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594212 4935 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:08:27.596970 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594215 4935 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:08:27.596970 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594217 4935 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:08:27.596970 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594220 4935 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:08:27.596970 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594223 4935 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:08:27.596970 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594226 4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:08:27.596970 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594229 4935 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:08:27.596970 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594231 4935 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:08:27.596970 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594234 4935 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:08:27.596970 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594237 4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:08:27.596970 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594240 4935 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:08:27.596970 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594242 4935 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:08:27.597519 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594245 4935 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:08:27.597519 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594249 4935 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:08:27.597519 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594251 4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:08:27.597519 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594254 4935 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:08:27.597519 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594256 4935 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:08:27.597519 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594259 4935 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:08:27.597519 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594261 4935 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:08:27.597519 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594264 4935 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:08:27.597519 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594266 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:08:27.597519 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594269 4935 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:08:27.597519 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594272 4935 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:08:27.597519 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594275 4935 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:08:27.597519 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594277 4935 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:08:27.597519 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594280 4935 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:08:27.597519 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594283 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:08:27.597519 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594285 4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:08:27.597519 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594288 4935 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:08:27.597519 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594291 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:08:27.597519 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594293 4935 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:08:27.598007 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594296 4935 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:08:27.598007 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594298 4935 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:08:27.598007 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594301 4935 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:08:27.598007 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594304 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:08:27.598007 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594306 4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:08:27.598007 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594309 4935 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:08:27.598007 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594311 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:08:27.598007 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594314 4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:08:27.598007 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594316 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:08:27.598007 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594319 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:08:27.598007 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594323 4935 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:08:27.598007 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594327 4935 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:08:27.598007 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594330 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:08:27.598007 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594333 4935 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:08:27.598007 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594337 4935 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:08:27.598007 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594940 4935 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:08:27.598007 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594949 4935 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:08:27.598007 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594951 4935 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:08:27.598007 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594955 4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:08:27.598515 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594957 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:08:27.598515 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594960 4935 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:08:27.598515 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594963 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:08:27.598515 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594966 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:08:27.598515 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594969 4935 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:08:27.598515 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594972 4935 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:08:27.598515 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594975 4935 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:08:27.598515 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594978 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:08:27.598515 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594980 4935 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:08:27.598515 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594983 4935 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:08:27.598515 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594986 4935 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:08:27.598515 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594989 4935 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:08:27.598515 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594991 4935 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:08:27.598515 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594994 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:08:27.598515 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594996 4935 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:08:27.598515 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.594999 4935 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:08:27.598515 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595002 4935 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:08:27.598515 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595005 4935 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:08:27.598515 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595007 4935 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:08:27.598515 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595010 4935 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:08:27.599036 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595012 4935 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:08:27.599036 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595015 4935 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:08:27.599036 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595017 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:08:27.599036 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595020 4935 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:08:27.599036 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595024 4935 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:08:27.599036 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595026 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:08:27.599036 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595029 4935 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:08:27.599036 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595032 4935 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:08:27.599036 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595034 4935 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:08:27.599036 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595037 4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:08:27.599036 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595040 4935 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:08:27.599036 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595044 4935 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:08:27.599036 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595049 4935 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:08:27.599036 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595052 4935 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:08:27.599036 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595055 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:08:27.599036 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595058 4935 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:08:27.599036 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595061 4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:08:27.599036 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595063 4935 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:08:27.599036 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595066 4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:08:27.599541 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595069 4935 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:08:27.599541 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595071 4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:08:27.599541 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595074 4935 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:08:27.599541 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595076 4935 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:08:27.599541 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595079 4935 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:08:27.599541 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595082 4935 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:08:27.599541 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595085 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:08:27.599541 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595087 4935 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:08:27.599541 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595090 4935 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:08:27.599541 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595092 4935 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:08:27.599541 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595096 4935 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:08:27.599541 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595098 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:08:27.599541 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595101 4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:08:27.599541 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595104 4935 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:08:27.599541 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595106 4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:08:27.599541 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595109 4935 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:08:27.599541 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595112 4935 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:08:27.599541 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595115 4935 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:08:27.599541 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595118 4935 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:08:27.599541 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595121 4935 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:08:27.600035 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595123 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:08:27.600035 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595126 4935 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:08:27.600035 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595141 4935 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:08:27.600035 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595144 4935 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:08:27.600035 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595146 4935 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:08:27.600035 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595152 4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:08:27.600035 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595155 4935 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:08:27.600035 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595158 4935 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:08:27.600035 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595161 4935 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:08:27.600035 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595163 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:08:27.600035 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595167 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:08:27.600035 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595171 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:08:27.600035 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595175 4935 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:08:27.600035 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595180 4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:08:27.600035 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595184 4935 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:08:27.600035 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595188 4935 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:08:27.600035 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595192 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:08:27.600035 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595195 4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:08:27.600035 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595198 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:08:27.600538 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595201 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:08:27.600538 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595203 4935 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:08:27.600538 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595206 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:08:27.600538 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595209 4935 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:08:27.600538 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595211 4935 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:08:27.600538 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595214 4935 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:08:27.600538 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595217 4935 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:08:27.600538 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595219 4935 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:08:27.600538 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595222 4935 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:08:27.600538 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595224 4935 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:08:27.600538 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595227 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:08:27.600538 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595229 4935 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:08:27.600538 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595237 4935 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:08:27.600538 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595240 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:08:27.600538 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595243 4935 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:08:27.600538 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595246 4935 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:08:27.600538 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595249 4935 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:08:27.600538 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595254 4935 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:08:27.600538 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595259 4935 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:08:27.600538 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595263 4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:08:27.601061 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595266 4935 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:08:27.601061 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595268 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:08:27.601061 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595271 4935 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:08:27.601061 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.595274 4935 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:08:27.601061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595374 4935 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 21:08:27.601061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595388 4935 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 21:08:27.601061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595395 4935 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 21:08:27.601061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595400 4935 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 21:08:27.601061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595405 4935 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 21:08:27.601061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595408 4935 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 21:08:27.601061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595413 4935 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 21:08:27.601061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595417 4935 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 21:08:27.601061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595420 4935 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 21:08:27.601061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595423 4935 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 21:08:27.601061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595427 4935 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 21:08:27.601061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595430 4935 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 21:08:27.601061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595434 4935 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 21:08:27.601061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595437 4935 flags.go:64] FLAG: --cgroup-root="" Nov 02 21:08:27.601061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595440 4935 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 21:08:27.601061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595443 4935 flags.go:64] FLAG: --client-ca-file="" Nov 02 21:08:27.601061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595446 4935 flags.go:64] FLAG: --cloud-provider="external" Nov 02 21:08:27.601061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595449 4935 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 21:08:27.601061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595453 4935 flags.go:64] FLAG: --cluster-domain="" Nov 02 21:08:27.601647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595456 4935 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 21:08:27.601647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595459 4935 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 21:08:27.601647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595462 4935 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 21:08:27.601647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595466 4935 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 21:08:27.601647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595471 4935 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 21:08:27.601647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595474 4935 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 21:08:27.601647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595477 4935 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 21:08:27.601647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595483 4935 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 21:08:27.601647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595486 4935 flags.go:64] FLAG: --contention-profiling="false" Nov 02 21:08:27.601647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595489 4935 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 21:08:27.601647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595492 4935 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 21:08:27.601647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595495 4935 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 21:08:27.601647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595498 4935 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 21:08:27.601647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595503 4935 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 21:08:27.601647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595506 4935 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 21:08:27.601647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595510 4935 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 21:08:27.601647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595513 4935 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 21:08:27.601647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595516 4935 flags.go:64] FLAG: --enable-server="true" Nov 02 21:08:27.601647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595519 4935 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 21:08:27.601647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595524 4935 flags.go:64] FLAG: --event-burst="100" Nov 02 21:08:27.601647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595527 4935 flags.go:64] FLAG: --event-qps="50" Nov 02 21:08:27.601647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595530 4935 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 21:08:27.601647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595533 4935 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 21:08:27.601647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595536 4935 flags.go:64] FLAG: --eviction-hard="" Nov 02 21:08:27.602424 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595541 4935 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 21:08:27.602424 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595544 4935 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 21:08:27.602424 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595548 4935 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 21:08:27.602424 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595552 4935 flags.go:64] FLAG: --eviction-soft="" Nov 02 21:08:27.602424 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595555 4935 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 21:08:27.602424 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595558 4935 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 21:08:27.602424 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595561 4935 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 21:08:27.602424 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595564 4935 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 21:08:27.602424 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595567 4935 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 21:08:27.602424 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595570 4935 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 21:08:27.602424 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595573 4935 flags.go:64] FLAG: --feature-gates="" Nov 02 21:08:27.602424 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595579 4935 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 21:08:27.602424 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595582 4935 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 21:08:27.602424 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595586 4935 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 21:08:27.602424 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595589 4935 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 21:08:27.602424 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595593 4935 flags.go:64] FLAG: --healthz-port="10248" Nov 02 21:08:27.602424 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595597 4935 flags.go:64] FLAG: --help="false" Nov 02 21:08:27.602424 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595600 4935 flags.go:64] FLAG: --hostname-override="ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:08:27.602424 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595603 4935 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 21:08:27.602424 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595606 4935 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 21:08:27.602424 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595609 4935 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 21:08:27.602424 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595613 4935 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 21:08:27.602424 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595616 4935 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 21:08:27.603016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595619 4935 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 21:08:27.603016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595622 4935 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 21:08:27.603016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595625 4935 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 21:08:27.603016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595628 4935 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 21:08:27.603016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595631 4935 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 21:08:27.603016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595635 4935 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 21:08:27.603016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595638 4935 flags.go:64] FLAG: --kube-reserved="" Nov 02 21:08:27.603016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595641 4935 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 21:08:27.603016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595643 4935 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 21:08:27.603016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595647 4935 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 21:08:27.603016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595650 4935 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 21:08:27.603016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595655 4935 flags.go:64] FLAG: --lock-file="" Nov 02 21:08:27.603016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595658 4935 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 21:08:27.603016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595661 4935 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 21:08:27.603016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595664 4935 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 21:08:27.603016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595669 4935 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 21:08:27.603016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595672 4935 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 21:08:27.603016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595675 4935 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 21:08:27.603016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595678 4935 flags.go:64] FLAG: --logging-format="text" Nov 02 21:08:27.603016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595681 4935 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 21:08:27.603016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595685 4935 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 21:08:27.603016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595689 4935 flags.go:64] FLAG: --manifest-url="" Nov 02 21:08:27.603016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595692 4935 flags.go:64] FLAG: --manifest-url-header="" Nov 02 21:08:27.603016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595696 4935 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 21:08:27.603016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595702 4935 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 21:08:27.603677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595706 4935 flags.go:64] FLAG: --max-pods="110" Nov 02 21:08:27.603677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595709 4935 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 21:08:27.603677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595712 4935 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 21:08:27.603677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595715 4935 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 21:08:27.603677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595718 4935 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 21:08:27.603677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595721 4935 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 21:08:27.603677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595724 4935 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 21:08:27.603677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595727 4935 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 21:08:27.603677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595736 4935 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 21:08:27.603677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595740 4935 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 21:08:27.603677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595743 4935 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 21:08:27.603677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595746 4935 flags.go:64] FLAG: --pod-cidr="" Nov 02 21:08:27.603677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595749 4935 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 21:08:27.603677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595752 4935 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 21:08:27.603677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595755 4935 flags.go:64] FLAG: --pods-per-core="0" Nov 02 21:08:27.603677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595758 4935 flags.go:64] FLAG: --port="10250" Nov 02 21:08:27.603677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595761 4935 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 21:08:27.603677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595764 4935 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0ce983b9fe3d8064f" Nov 02 21:08:27.603677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595769 4935 flags.go:64] FLAG: --qos-reserved="" Nov 02 21:08:27.603677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595772 4935 flags.go:64] FLAG: --read-only-port="10255" Nov 02 21:08:27.603677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595775 4935 flags.go:64] FLAG: --register-node="true" Nov 02 21:08:27.603677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595778 4935 flags.go:64] FLAG: --register-with-taints="" Nov 02 21:08:27.603677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595782 4935 flags.go:64] FLAG: --registry-burst="10" Nov 02 21:08:27.603677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595785 4935 flags.go:64] FLAG: --registry-qps="5" Nov 02 21:08:27.603677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595788 4935 flags.go:64] FLAG: --reserved-cpus="" Nov 02 21:08:27.603677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595791 4935 flags.go:64] FLAG: --reserved-memory="" Nov 02 21:08:27.604368 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595796 4935 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 21:08:27.604368 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595799 4935 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 21:08:27.604368 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595802 4935 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 21:08:27.604368 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595808 4935 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 21:08:27.604368 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595811 4935 flags.go:64] FLAG: --runonce="false" Nov 02 21:08:27.604368 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595815 4935 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 21:08:27.604368 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595818 4935 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 21:08:27.604368 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595821 4935 flags.go:64] FLAG: --seccomp-default="false" Nov 02 21:08:27.604368 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595825 4935 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 21:08:27.604368 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595828 4935 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 21:08:27.604368 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595831 4935 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 21:08:27.604368 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595834 4935 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 21:08:27.604368 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595837 4935 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 21:08:27.604368 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595841 4935 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 21:08:27.604368 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595844 4935 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 21:08:27.604368 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595847 4935 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 21:08:27.604368 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595850 4935 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 21:08:27.604368 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595853 4935 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 21:08:27.604368 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595856 4935 flags.go:64] FLAG: --system-cgroups="" Nov 02 21:08:27.604368 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595859 4935 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 21:08:27.604368 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595864 4935 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 21:08:27.604368 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595867 4935 flags.go:64] FLAG: --tls-cert-file="" Nov 02 21:08:27.604368 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595870 4935 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 21:08:27.604368 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595874 4935 flags.go:64] FLAG: --tls-min-version="" Nov 02 21:08:27.604368 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595878 4935 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 21:08:27.604982 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595881 4935 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 21:08:27.604982 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595884 4935 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 21:08:27.604982 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595887 4935 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 21:08:27.604982 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595890 4935 flags.go:64] FLAG: --v="2" Nov 02 21:08:27.604982 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595894 4935 flags.go:64] FLAG: --version="false" Nov 02 21:08:27.604982 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595898 4935 flags.go:64] FLAG: --vmodule="" Nov 02 21:08:27.604982 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595902 4935 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 21:08:27.604982 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.595905 4935 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 21:08:27.604982 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596064 4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:08:27.604982 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596069 4935 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:08:27.604982 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596072 4935 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:08:27.604982 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596075 4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:08:27.604982 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596079 4935 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:08:27.604982 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596082 4935 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:08:27.604982 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596085 4935 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:08:27.604982 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596088 4935 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:08:27.604982 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596090 4935 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:08:27.604982 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596093 4935 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:08:27.604982 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596095 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:08:27.604982 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596098 4935 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:08:27.604982 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596102 4935 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:08:27.605521 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596105 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:08:27.605521 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596108 4935 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:08:27.605521 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596111 4935 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:08:27.605521 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596114 4935 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:08:27.605521 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596117 4935 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:08:27.605521 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596120 4935 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:08:27.605521 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596122 4935 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:08:27.605521 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596125 4935 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:08:27.605521 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596141 4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:08:27.605521 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596144 4935 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:08:27.605521 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596151 4935 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:08:27.605521 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596154 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:08:27.605521 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596156 4935 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:08:27.605521 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596159 4935 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:08:27.605521 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596162 4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:08:27.605521 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596165 4935 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:08:27.605521 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596168 4935 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:08:27.605521 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596171 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:08:27.605521 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596173 4935 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:08:27.605521 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596176 4935 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:08:27.606026 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596179 4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:08:27.606026 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596182 4935 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:08:27.606026 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596184 4935 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:08:27.606026 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596189 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:08:27.606026 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596192 4935 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:08:27.606026 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596194 4935 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:08:27.606026 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596197 4935 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:08:27.606026 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596200 4935 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:08:27.606026 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596202 4935 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:08:27.606026 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596205 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:08:27.606026 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596207 4935 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:08:27.606026 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596210 4935 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:08:27.606026 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596212 4935 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:08:27.606026 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596215 4935 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:08:27.606026 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596217 4935 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:08:27.606026 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596220 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:08:27.606026 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596223 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:08:27.606026 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596225 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:08:27.606026 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596228 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:08:27.606542 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596230 4935 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:08:27.606542 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596234 4935 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:08:27.606542 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596239 4935 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:08:27.606542 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596244 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:08:27.606542 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596247 4935 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:08:27.606542 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596250 4935 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:08:27.606542 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596253 4935 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:08:27.606542 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596256 4935 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:08:27.606542 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596259 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:08:27.606542 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596261 4935 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:08:27.606542 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596265 4935 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:08:27.606542 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596267 4935 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:08:27.606542 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596270 4935 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:08:27.606542 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596273 4935 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:08:27.606542 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596275 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:08:27.606542 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596278 4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:08:27.606542 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596282 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:08:27.606542 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596285 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:08:27.606542 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596288 4935 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:08:27.607096 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596291 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:08:27.607096 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596293 4935 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:08:27.607096 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596296 4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:08:27.607096 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596299 4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:08:27.607096 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596301 4935 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:08:27.607096 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596304 4935 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:08:27.607096 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596307 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:08:27.607096 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596310 4935 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:08:27.607096 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596312 4935 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:08:27.607096 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596315 4935 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:08:27.607096 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596318 4935 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:08:27.607096 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596320 4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:08:27.607096 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596323 4935 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:08:27.607096 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596351 4935 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:08:27.607096 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596355 4935 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:08:27.607096 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596357 4935 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:08:27.607096 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596362 4935 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:08:27.607096 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596365 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:08:27.607096 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596368 4935 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:08:27.607096 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596370 4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:08:27.607847 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596373 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:08:27.607847 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596376 4935 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:08:27.607847 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596378 4935 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:08:27.607847 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596381 4935 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:08:27.607847 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596384 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:08:27.607847 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596386 4935 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:08:27.607847 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596389 4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:08:27.607847 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596391 4935 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:08:27.607847 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596394 4935 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:08:27.607847 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596398 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:08:27.607847 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596400 4935 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:08:27.607847 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596404 4935 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:08:27.607847 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596407 4935 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:08:27.607847 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596410 4935 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:08:27.607847 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596412 4935 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:08:27.607847 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.596442 4935 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:08:27.608347 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596485 4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:08:27.608347 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596491 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:08:27.608347 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596494 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:08:27.608347 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596497 4935 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:08:27.608347 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596500 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:08:27.608347 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596502 4935 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:08:27.608347 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596505 4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:08:27.608347 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596508 4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:08:27.608347 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596510 4935 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:08:27.608347 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596514 4935 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:08:27.608347 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596519 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:08:27.608347 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596523 4935 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:08:27.608347 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596528 4935 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:08:27.608347 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596533 4935 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:08:27.608347 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596537 4935 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:08:27.608347 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596542 4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:08:27.608347 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596545 4935 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:08:27.608347 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596548 4935 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:08:27.608347 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596550 4935 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:08:27.608347 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596553 4935 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:08:27.609051 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596556 4935 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:08:27.609051 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596558 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:08:27.609051 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596561 4935 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:08:27.609051 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596563 4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:08:27.609051 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596566 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:08:27.609051 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596569 4935 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:08:27.609051 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596591 4935 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:08:27.609051 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596594 4935 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:08:27.609051 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596597 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:08:27.609051 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596601 4935 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:08:27.609051 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596603 4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:08:27.609051 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596606 4935 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:08:27.609051 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596609 4935 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:08:27.609051 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596611 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:08:27.609051 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596614 4935 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:08:27.609051 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596616 4935 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:08:27.609051 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596619 4935 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:08:27.609051 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596622 4935 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:08:27.609051 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596624 4935 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:08:27.609557 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596627 4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:08:27.609557 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596629 4935 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:08:27.609557 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596632 4935 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:08:27.609557 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596635 4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:08:27.609557 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596637 4935 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:08:27.609557 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596641 4935 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:08:27.609557 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596644 4935 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:08:27.609557 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596647 4935 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:08:27.609557 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596649 4935 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:08:27.609557 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596652 4935 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:08:27.609557 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596655 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:08:27.609557 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596657 4935 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:08:27.609557 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596661 4935 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:08:27.609557 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596665 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:08:27.609557 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596667 4935 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:08:27.609557 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596670 4935 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:08:27.609557 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596673 4935 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:08:27.609557 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596676 4935 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:08:27.609557 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596679 4935 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:08:27.610016 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596682 4935 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:08:27.610016 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596684 4935 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:08:27.610016 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596687 4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:08:27.610016 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596690 4935 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:08:27.610016 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596692 4935 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:08:27.610016 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596695 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:08:27.610016 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596697 4935 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:08:27.610016 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596700 4935 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:08:27.610016 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596702 4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:08:27.610016 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596705 4935 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:08:27.610016 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596707 4935 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:08:27.610016 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596710 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:08:27.610016 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596712 4935 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:08:27.610016 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596715 4935 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:08:27.610016 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596717 4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:08:27.610016 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596720 4935 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:08:27.610016 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596723 4935 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:08:27.610016 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596725 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:08:27.610016 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596729 4935 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:08:27.610016 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596731 4935 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:08:27.610661 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596734 4935 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:08:27.610661 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596736 4935 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:08:27.610661 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596739 4935 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:08:27.610661 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596741 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:08:27.610661 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596744 4935 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:08:27.610661 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596746 4935 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:08:27.610661 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596749 4935 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:08:27.610661 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596752 4935 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:08:27.610661 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596754 4935 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:08:27.610661 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596757 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:08:27.610661 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596759 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:08:27.610661 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596761 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:08:27.610661 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596764 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:08:27.610661 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596766 4935 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:08:27.610661 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596770 4935 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:08:27.610661 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596775 4935 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:08:27.610661 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596778 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:08:27.610661 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596782 4935 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:08:27.610661 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596784 4935 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:08:27.611287 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596787 4935 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:08:27.611287 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596789 4935 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:08:27.611287 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596792 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:08:27.611287 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596794 4935 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:08:27.611287 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596797 4935 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:08:27.611287 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596800 4935 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:08:27.611287 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596802 4935 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:08:27.611287 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596805 4935 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:08:27.611287 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.596807 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:08:27.611287 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.601094 4935 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 21:08:27.611287 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.601110 4935 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 21:08:27.611287 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.601138 4935 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 21:08:27.611287 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.601145 4935 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 21:08:27.611287 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601211 4935 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:08:27.611287 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601217 4935 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:08:27.611287 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601220 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:08:27.611287 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601223 4935 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:08:27.611287 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601227 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:08:27.611287 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601230 4935 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:08:27.611788 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601232 4935 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:08:27.611788 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601235 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:08:27.611788 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601238 4935 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:08:27.611788 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601240 4935 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:08:27.611788 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601243 4935 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:08:27.611788 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601245 4935 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:08:27.611788 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601248 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:08:27.611788 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601250 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:08:27.611788 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601253 4935 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:08:27.611788 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601255 4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:08:27.611788 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601266 4935 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:08:27.611788 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601271 4935 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:08:27.611788 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601274 4935 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:08:27.611788 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601277 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:08:27.611788 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601280 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:08:27.611788 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601283 4935 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:08:27.611788 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601287 4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:08:27.611788 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601289 4935 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:08:27.611788 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601292 4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:08:27.612389 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601294 4935 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:08:27.612389 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601297 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:08:27.612389 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601300 4935 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:08:27.612389 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601302 4935 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:08:27.612389 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601305 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:08:27.612389 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601308 4935 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:08:27.612389 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601310 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:08:27.612389 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601313 4935 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:08:27.612389 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601315 4935 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:08:27.612389 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601318 4935 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:08:27.612389 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601320 4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:08:27.612389 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601323 4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:08:27.612389 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601326 4935 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:08:27.612389 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601328 4935 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:08:27.612389 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601331 4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:08:27.612389 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601334 4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:08:27.612389 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601336 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:08:27.612389 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601339 4935 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:08:27.612389 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601342 4935 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:08:27.612389 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601344 4935 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:08:27.612894 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601347 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:08:27.612894 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601349 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:08:27.612894 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601352 4935 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:08:27.612894 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601355 4935 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:08:27.612894 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601366 4935 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:08:27.612894 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601369 4935 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:08:27.612894 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601372 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:08:27.612894 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601375 4935 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:08:27.612894 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601377 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:08:27.612894 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601380 4935 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:08:27.612894 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601383 4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:08:27.612894 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601385 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:08:27.612894 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601388 4935 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:08:27.612894 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601391 4935 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:08:27.612894 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601393 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:08:27.612894 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601396 4935 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:08:27.612894 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601398 4935 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:08:27.612894 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601401 4935 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:08:27.612894 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601404 4935 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:08:27.613381 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601406 4935 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:08:27.613381 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601409 4935 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:08:27.613381 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601412 4935 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:08:27.613381 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601414 4935 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:08:27.613381 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601416 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:08:27.613381 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601419 4935 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:08:27.613381 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601422 4935 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:08:27.613381 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601424 4935 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:08:27.613381 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601427 4935 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:08:27.613381 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601429 4935 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:08:27.613381 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601432 4935 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:08:27.613381 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601434 4935 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:08:27.613381 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601437 4935 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:08:27.613381 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601439 4935 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:08:27.613381 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601443 4935 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:08:27.613381 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601446 4935 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:08:27.613381 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601449 4935 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:08:27.613381 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601452 4935 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:08:27.613381 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601460 4935 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:08:27.613893 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601463 4935 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:08:27.613893 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601466 4935 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:08:27.613893 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601469 4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:08:27.613893 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601471 4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:08:27.613893 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601474 4935 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:08:27.613893 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601476 4935 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:08:27.613893 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601479 4935 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:08:27.613893 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601482 4935 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:08:27.613893 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601484 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:08:27.613893 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601487 4935 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:08:27.613893 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601489 4935 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:08:27.613893 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601492 4935 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:08:27.613893 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601494 4935 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:08:27.613893 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601497 4935 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:08:27.613893 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601500 4935 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:08:27.613893 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601502 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:08:27.613893 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601505 4935 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:08:27.613893 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601507 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:08:27.613893 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601510 4935 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:08:27.613893 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601512 4935 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:08:27.614573 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601519 4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:08:27.614573 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601522 4935 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:08:27.614573 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.601525 4935 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:08:27.614573 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.601556 4935 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:08:27.614573 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.602980 4935 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:08:27.614573 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603021 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:08:27.614573 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603026 4935 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:08:27.614573 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603029 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:08:27.614573 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603035 4935 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:08:27.614573 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603075 4935 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:08:27.614573 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603092 4935 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:08:27.614573 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603098 4935 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:08:27.614573 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603103 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:08:27.614573 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603109 4935 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:08:27.614573 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603114 4935 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:08:27.614573 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603118 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:08:27.614971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603125 4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:08:27.614971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603144 4935 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:08:27.614971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603150 4935 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:08:27.614971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603154 4935 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:08:27.614971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603160 4935 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:08:27.614971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603163 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:08:27.614971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603169 4935 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:08:27.614971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603172 4935 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:08:27.614971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603176 4935 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:08:27.614971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603180 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:08:27.614971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603184 4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:08:27.614971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603187 4935 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:08:27.614971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603190 4935 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:08:27.614971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603193 4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:08:27.614971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603212 4935 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:08:27.614971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603344 4935 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:08:27.614971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603353 4935 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:08:27.614971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603358 4935 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:08:27.614971 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603362 4935 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:08:27.615450 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603366 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:08:27.615450 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603371 4935 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:08:27.615450 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603376 4935 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:08:27.615450 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603380 4935 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:08:27.615450 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603384 4935 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:08:27.615450 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603390 4935 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:08:27.615450 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603394 4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:08:27.615450 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603398 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:08:27.615450 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603402 4935 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:08:27.615450 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603408 4935 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:08:27.615450 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603413 4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:08:27.615450 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603417 4935 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:08:27.615450 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603421 4935 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:08:27.615450 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603425 4935 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:08:27.615450 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603429 4935 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:08:27.615450 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603434 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:08:27.615450 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603439 4935 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:08:27.615450 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603443 4935 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:08:27.615450 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603447 4935 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:08:27.615450 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603451 4935 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:08:27.615924 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603455 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:08:27.615924 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603459 4935 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:08:27.615924 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603463 4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:08:27.615924 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603467 4935 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:08:27.615924 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603471 4935 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:08:27.615924 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603475 4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:08:27.615924 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603479 4935 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:08:27.615924 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603486 4935 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:08:27.615924 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603494 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:08:27.615924 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603500 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:08:27.615924 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603504 4935 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:08:27.615924 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603509 4935 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:08:27.615924 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603513 4935 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:08:27.615924 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603517 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:08:27.615924 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603522 4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:08:27.615924 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603526 4935 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:08:27.615924 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603530 4935 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:08:27.615924 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603535 4935 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:08:27.615924 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603539 4935 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:08:27.616561 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603543 4935 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:08:27.616561 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603548 4935 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:08:27.616561 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603552 4935 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:08:27.616561 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603557 4935 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:08:27.616561 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603561 4935 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:08:27.616561 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603565 4935 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:08:27.616561 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603569 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:08:27.616561 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603573 4935 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:08:27.616561 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603578 4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:08:27.616561 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603582 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:08:27.616561 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603585 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:08:27.616561 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603587 4935 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:08:27.616561 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603590 4935 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:08:27.616561 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603593 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:08:27.616561 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603596 4935 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:08:27.616561 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603598 4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:08:27.616561 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603601 4935 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:08:27.616561 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603603 4935 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:08:27.616561 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603606 4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:08:27.616561 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603609 4935 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:08:27.617070 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603612 4935 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:08:27.617070 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603614 4935 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:08:27.617070 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603617 4935 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:08:27.617070 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603620 4935 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:08:27.617070 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603622 4935 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:08:27.617070 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603625 4935 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:08:27.617070 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603629 4935 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:08:27.617070 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603631 4935 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:08:27.617070 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603634 4935 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:08:27.617070 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603637 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:08:27.617070 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603639 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:08:27.617070 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603642 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:08:27.617070 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603645 4935 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:08:27.617070 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603649 4935 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:08:27.617070 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603653 4935 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:08:27.617070 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603656 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:08:27.617480 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.603690 4935 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:08:27.617480 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603733 4935 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:08:27.617480 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603738 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:08:27.617480 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603741 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:08:27.617480 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603744 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:08:27.617480 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603748 4935 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:08:27.617480 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603752 4935 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:08:27.617480 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603756 4935 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:08:27.617480 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603759 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:08:27.617480 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603763 4935 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:08:27.617480 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603767 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:08:27.617480 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603771 4935 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:08:27.617480 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603774 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:08:27.617480 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603777 4935 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:08:27.617480 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603780 4935 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:08:27.617855 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603783 4935 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:08:27.617855 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603786 4935 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:08:27.617855 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603788 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:08:27.617855 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603791 4935 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:08:27.617855 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603794 4935 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:08:27.617855 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603796 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:08:27.617855 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603799 4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:08:27.617855 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603802 4935 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:08:27.617855 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603804 4935 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:08:27.617855 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603807 4935 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:08:27.617855 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603810 4935 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:08:27.617855 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603812 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:08:27.617855 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603815 4935 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:08:27.617855 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603818 4935 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:08:27.617855 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603820 4935 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:08:27.617855 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603823 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:08:27.617855 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603826 4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:08:27.617855 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603828 4935 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:08:27.617855 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603831 4935 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:08:27.618354 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603833 4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:08:27.618354 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603836 4935 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:08:27.618354 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603838 4935 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:08:27.618354 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603841 4935 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:08:27.618354 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603845 4935 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:08:27.618354 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603847 4935 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:08:27.618354 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603850 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:08:27.618354 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603852 4935 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:08:27.618354 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603855 4935 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:08:27.618354 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603858 4935 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:08:27.618354 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603861 4935 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:08:27.618354 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603864 4935 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:08:27.618354 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603866 4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:08:27.618354 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603869 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:08:27.618354 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603871 4935 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:08:27.618354 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603874 4935 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:08:27.618354 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603877 4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:08:27.618354 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603879 4935 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:08:27.618354 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603882 4935 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:08:27.618354 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603884 4935 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:08:27.618862 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603887 4935 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:08:27.618862 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603889 4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:08:27.618862 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603892 4935 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:08:27.618862 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603894 4935 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:08:27.618862 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603898 4935 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:08:27.618862 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603900 4935 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:08:27.618862 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603903 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:08:27.618862 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603905 4935 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:08:27.618862 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603908 4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:08:27.618862 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603911 4935 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:08:27.618862 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603913 4935 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:08:27.618862 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603916 4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:08:27.618862 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603918 4935 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:08:27.618862 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603923 4935 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:08:27.618862 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603927 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:08:27.618862 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603931 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:08:27.618862 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603933 4935 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:08:27.618862 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603936 4935 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:08:27.618862 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603939 4935 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:08:27.619357 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603942 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:08:27.619357 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603944 4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:08:27.619357 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603947 4935 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:08:27.619357 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603950 4935 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:08:27.619357 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603953 4935 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:08:27.619357 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603955 4935 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:08:27.619357 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603958 4935 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:08:27.619357 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603961 4935 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:08:27.619357 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603963 4935 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:08:27.619357 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603966 4935 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:08:27.619357 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603969 4935 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:08:27.619357 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603971 4935 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:08:27.619357 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603974 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:08:27.619357 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603976 4935 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:08:27.619357 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603979 4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:08:27.619357 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603981 4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:08:27.619357 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603984 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:08:27.619357 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603987 4935 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:08:27.619357 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603989 4935 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:08:27.619357 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603992 4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:08:27.620179 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603995 4935 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:08:27.620179 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.603997 4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:08:27.620179 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.604000 4935 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:08:27.620179 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.604003 4935 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:08:27.620179 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.604005 4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:08:27.620179 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.604008 4935 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:08:27.620179 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.604010 4935 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:08:27.620179 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.604013 4935 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:08:27.620179 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.604015 4935 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:08:27.620179 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.604018 4935 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:08:27.620179 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.604021 4935 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:08:27.620179 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.604024 4935 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:08:27.620179 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.604027 4935 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:08:27.620179 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:27.604030 4935 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:08:27.620179 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.604089 4935 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 21:08:27.620179 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.605069 4935 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 21:08:27.620179 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.605148 4935 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 21:08:27.620179 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.605365 4935 server.go:1014] "Starting client certificate rotation" Nov 02 21:08:27.620179 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.605497 4935 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 21:08:27.620853 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.605540 4935 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 21:02:45 +0000 UTC" deadline="2028-05-30 17:25:18.266438405 +0000 UTC" Nov 02 21:08:27.620853 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.605567 4935 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13796h16m50.660873724s" Nov 02 21:08:27.620853 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.605826 4935 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:08:27.620853 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.606921 4935 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:08:27.620853 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.608507 4935 log.go:25] "Validated CRI v1 runtime API" Nov 02 21:08:27.620853 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.610181 4935 log.go:25] "Validated CRI v1 image API" Nov 02 21:08:27.620853 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.610998 4935 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 21:08:27.620853 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.612913 4935 fs.go:135] Filesystem UUIDs: map[387dbd0c-85bc-41c1-b895-d36e9307ce67:/dev/nvme0n1p4 5f8ee96b-f322-4201-ba54-0cca4207be19:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 02 21:08:27.621109 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.612928 4935 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01c2aa740e152759c0ae026d7d14853bd1ce15949ee48d84b6c1118358ec89dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01c2aa740e152759c0ae026d7d14853bd1ce15949ee48d84b6c1118358ec89dd/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ebda27314aede86920f44c514e1aeb2348dcaf88ab8704a9d68111a733f1b16/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ebda27314aede86920f44c514e1aeb2348dcaf88ab8704a9d68111a733f1b16/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24cf1e198be9f0535676921dffad7301a8eec044161ab4b7507aa1fe58f1dc6d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24cf1e198be9f0535676921dffad7301a8eec044161ab4b7507aa1fe58f1dc6d/userdata/shm major:0 minor:341 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/26a2e1e33242c28dc251c800d13b9c6c302e9c1d7705af705aea7e307fc83ec6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/26a2e1e33242c28dc251c800d13b9c6c302e9c1d7705af705aea7e307fc83ec6/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d12fd879df50acc9c58da47ef0db58f3bb45a29369edbe0f5eb5b196f9dec48/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d12fd879df50acc9c58da47ef0db58f3bb45a29369edbe0f5eb5b196f9dec48/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/43eabb098ccce10b169118e5fb17731ed32532e6e9057a0359575b38e98c8932/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/43eabb098ccce10b169118e5fb17731ed32532e6e9057a0359575b38e98c8932/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46114be710d5a4afe8f3e6fb9d6da86b2d9882a4ad235f92ef80f3710c460d13/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46114be710d5a4afe8f3e6fb9d6da86b2d9882a4ad235f92ef80f3710c460d13/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/595bbca93c4ad043d8e398da486a2883ff60f00a9aaaaf8bcec628fde4087f03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/595bbca93c4ad043d8e398da486a2883ff60f00a9aaaaf8bcec628fde4087f03/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/67621105d0befe73aa74ba7e09f62ac325157669fa22233822aafe5b1fb981f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/67621105d0befe73aa74ba7e09f62ac325157669fa22233822aafe5b1fb981f2/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/77adee5b179ff360088237e14769bbde2edd21eab5ed31dca582284317167945/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/77adee5b179ff360088237e14769bbde2edd21eab5ed31dca582284317167945/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f401984caa01a6a03fd60b1964dcc5d3ff649eccb524d2222249b131590e1b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f401984caa01a6a03fd60b1964dcc5d3ff649eccb524d2222249b131590e1b8/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/904c2e78a563c743911689439b0018c42b7c9ae7c573061d777d617010d61203/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/904c2e78a563c743911689439b0018c42b7c9ae7c573061d777d617010d61203/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/99213b29ed642ee2d9ddc00d9043868fee9f471355a4de7ce5e4cb2a7e78f446/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/99213b29ed642ee2d9ddc00d9043868fee9f471355a4de7ce5e4cb2a7e78f446/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/996e3208f62bddecd270a96f63112c517e921203b4ff8047fa02bb22f44c06d1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/996e3208f62bddecd270a96f63112c517e921203b4ff8047fa02bb22f44c06d1/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a47e2648877ec7c53a35a6977dc0d961c05a330311daad6d70cfe41f3b51d198/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a47e2648877ec7c53a35a6977dc0d961c05a330311daad6d70cfe41f3b51d198/userdata/shm major:0 minor:258 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a523dc598caeeb5d5889543053a3cd0656321bcf0752eb039e61ec5e9c08e5d2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a523dc598caeeb5d5889543053a3cd0656321bcf0752eb039e61ec5e9c08e5d2/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b7afbd2ef8c2fd552a7bde8d073aae109ba33a010abf7b03f5689b7c6f7c8589/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b7afbd2ef8c2fd552a7bde8d073aae109ba33a010abf7b03f5689b7c6f7c8589/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d189fe9831262764a041fc7b52264ccceae18308ea1905cab9e7ce1a790e5ed7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d189fe9831262764a041fc7b52264ccceae18308ea1905cab9e7ce1a790e5ed7/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d5d4f6ec3bb69138dd6f297adbe5cc65f427c366c7e7e872caa1ea7bd8e267af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d5d4f6ec3bb69138dd6f297adbe5cc65f427c366c7e7e872caa1ea7bd8e267af/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dc462923466524e209c628f11d6b36ff1698809fd6c547d6b1821ccb8fb05c0d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dc462923466524e209c628f11d6b36ff1698809fd6c547d6b1821ccb8fb05c0d/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e8c8a4724dec7748c53ba6fa336407ba878e7ff4704c694f882fcce1127403de/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e8c8a4724dec7748c53ba6fa336407ba878e7ff4704c694f882fcce1127403de/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed611eb381e81907dcc1e8778781305f8612bdde748f9c61c70645463827e131/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed611eb381e81907dcc1e8778781305f8612bdde748f9c61c70645463827e131/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fea13107742b81caf647af1279a23748237c82739f6b840f2bb100c0be1127d1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fea13107742b81caf647af1279a23748237c82739f6b840f2bb100c0be1127d1/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08b38b87-8fd7-475e-94db-8bdd1339dc4b/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/08b38b87-8fd7-475e-94db-8bdd1339dc4b/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08b38b87-8fd7-475e-94db-8bdd1339dc4b/volumes/kubernetes.io~projected/kube-api-access-bxbh5:{mountpoint:/var/lib/kubelet/pods/08b38b87-8fd7-475e-94db-8bdd1339dc4b/volumes/kubernetes.io~projected/kube-api-access-bxbh5 major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08b38b87-8fd7-475e-94db-8bdd1339dc4b/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/08b38b87-8fd7-475e-94db-8bdd1339dc4b/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08b38b87-8fd7-475e-94db-8bdd1339dc4b/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/08b38b87-8fd7-475e-94db-8bdd1339dc4b/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/345370c7-0ca8-4afe-94db-c9f5394d4d72/volumes/kubernetes.io~projected/kube-api-access-rfhs6:{mountpoint:/var/lib/kubelet/pods/345370c7-0ca8-4afe-94db-c9f5394d4d72/volumes/kubernetes.io~projected/kube-api-access-rfhs6 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39830cb0-c259-446d-b2d3-fd8956b968fb/volumes/kubernetes.io~projected/kube-api-access-9kmdm:{mountpoint:/var/lib/kubelet/pods/39830cb0-c259-446d-b2d3-fd8956b968fb/volumes/kubernetes.io~projected/kube-api-access-9kmdm major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ab0662d-ef78-415d-98f6-8b271e23e16f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/3ab0662d-ef78-415d-98f6-8b271e23e16f/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42b83ba5-5e98-459f-ad4f-6a94c90212be/volumes/kubernetes.io~projected/kube-api-access-lq5t9:{mountpoint:/var/lib/kubelet/pods/42b83ba5-5e98-459f-ad4f-6a94c90212be/volumes/kubernetes.io~projected/kube-api-access-lq5t9 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44a209b7-2144-45a0-bea0-2f8adf6658cb/volumes/kubernetes.io~projected/kube-api-access-hxcnw:{mountpoint:/var/lib/kubelet/pods/44a209b7-2144-45a0-bea0-2f8adf6658cb/volumes/kubernetes.io~projected/kube-api-access-hxcnw major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44a209b7-2144-45a0-bea0-2f8adf6658cb/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/44a209b7-2144-45a0-bea0-2f8adf6658cb/volumes/kubernetes.io~secret/default-certificate major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44a209b7-2144-45a0-bea0-2f8adf6658cb/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/44a209b7-2144-45a0-bea0-2f8adf6658cb/volumes/kubernetes.io~secret/stats-auth major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ad2a39a-7a74-44a2-a97f-88e84e65616c/volumes/kubernetes.io~projected/kube-api-access-fmqww:{mountpoint:/var/lib/kubelet/pods/4ad2a39a-7a74-44a2-a97f-88e84e65616c/volumes/kubernetes.io~projected/kube-api-access-fmqww major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ca28f2b-c77d-4aa1-8501-ec380253ab88/volumes/kubernetes.io~projected/kube-api-access-t7kkw:{mountpoint:/var/lib/kubelet/pods/5ca28f2b-c77d-4aa1-8501-ec380253ab88/volumes/kubernetes.io~projected/kube-api-access-t7kkw major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ca28f2b-c77d-4aa1-8501-ec380253ab88/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/5ca28f2b-c77d-4aa1-8501-ec380253ab88/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67c3caa0-0b53-42c9-98b2-96addd3dd7fc/volumes/kubernetes.io~projected/kube-api-access-jx8dl:{mountpoint:/var/lib/kubelet/pods/67c3caa0-0b53-42c9-98b2-96addd3dd7fc/volumes/kubernetes.io~projected/kube-api-access-jx8dl major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b5655ce-43ca-48e4-9e4b-872283ccb85c/volumes/kubernetes.io~projected/kube-api-access-q672k:{mountpoint:/var/lib/kubelet/pods/6b5655ce-43ca-48e4-9e4b-872283ccb85c/volumes/kubernetes.io~projected/kube-api-access-q672k major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7dab1353-88e8-4250-b7ab-a4b2a69e37fa/volumes/kubernetes.io~projected/kube-api-access-wgqft:{mountpoint:/var/lib/kubelet/pods/7dab1353-88e8-4250-b7ab-a4b2a69e37fa/volumes/kubernetes.io~projected/kube-api-access-wgqft major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7de0fc42-d787-49e7-80a2-5122a67bfba0/volumes/kubernetes.io~projected/kube-api-access-jxm2l:{mountpoint:/var/lib/kubelet/pods/7de0fc42-d787-49e7-80a2-5122a67bfba0/volumes/kubernetes.io~projected/kube-api-access-jxm2l major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7de0fc42-d787-49e7-80a2-5122a67bfba0/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/7de0fc42-d787-49e7-80a2-5122a67bfba0/volumes/kubernetes.io~secret/cert major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ca0bae8-7539-4b19-912c-4379064ecfa2/volumes/kubernetes.io~projected/kube-api-access-gtj26:{mountpoint:/var/lib/kubelet/pods/8ca0bae8-7539-4b19-912c-4379064ecfa2/volumes/kubernetes.io~projected/kube-api-access-gtj26 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cf65e09-2079-40cc-b597-1b67ba482a04/volumes/kubernetes.io~projected/kube-api-access-5v2wr:{mountpoint:/var/lib/kubelet/pods/8cf65e09-2079-40cc-b597-1b67ba482a04/volumes/kubernetes.io~projected/kube-api-access-5v2wr major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cf65e09-2079-40cc-b597-1b67ba482a04/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8cf65e09-2079-40cc-b597-1b67ba482a04/volumes/kubernetes.io~secret/serving-cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/934bd6cf-3dcb-40fd-9b11-97886b3f098b/volumes/kubernetes.io~projected/kube-api-access-vx9qw:{mountpoint:/var/lib/kubelet/pods/934bd6cf-3dcb-40fd-9b11-97886b3f098b/volumes/kubernetes.io~projected/kube-api-access-vx9qw major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a2820a3-6d1e-42d0-8a89-0e47f870abc4/volumes/kubernetes.io~projected/kube-api-access-gnvnx:{mountpoint:/var/lib/kubelet/pods/9a2820a3-6d1e-42d0-8a89-0e47f870abc4/volumes/kubernetes.io~projected/kube-api-access-gnvnx major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a2820a3-6d1e-42d0-8a89-0e47f870abc4/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/9a2820a3-6d1e-42d0-8a89-0e47f870abc4/volumes/kubernetes.io~secret/serving-cert major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7204d98-9f92-46e6-9e6d-482371e0a3ee/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/a7204d98-9f92-46e6-9e6d-482371e0a3ee/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7204d98-9f92-46e6-9e6d-482371e0a3ee/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/a7204d98-9f92-46e6-9e6d-482371e0a3ee/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7204d98-9f92-46e6-9e6d-482371e0a3ee/volumes/kubernetes.io~projected/kube-api-access-sx42g:{mountpoint:/var/lib/kubelet/pods/a7204d98-9f92-46e6-9e6d-482371e0a3ee/volumes/kubernetes.io~projected/kube-api-access-sx42g major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a899c9c5-79b4-472d-942c-5e0c4c53e954/volumes/kubernetes.io~projected/kube-api-access-9gttn:{mountpoint:/var/lib/kubelet/pods/a899c9c5-79b4-472d-942c-5e0c4c53e954/volumes/kubernetes.io~projected/kube-api-access-9gttn major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be519c9c-2677-4c4a-99c5-c537317bda3e/volumes/kubernetes.io~projected/kube-api-access-fshk5:{mountpoint:/var/lib/kubelet/pods/be519c9c-2677-4c4a-99c5-c537317bda3e/volumes/kubernetes.io~projected/kube-api-access-fshk5 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bff8d097-26a3-4621-ac01-c3b00ed23e06/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/bff8d097-26a3-4621-ac01-c3b00ed23e06/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bff8d097-26a3-4621-ac01-c3b00ed23e06/volumes/kubernetes.io~projected/kube-api-access-w6wd7:{mountpoint:/var/lib/kubelet/pods/bff8d097-26a3-4621-ac01-c3b00ed23e06/volumes/kubernetes.io~projected/kube-api-access-w6wd7 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bff8d097-26a3-4621-ac01-c3b00ed23e06/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/bff8d097-26a3-4621-ac01-c3b00ed23e06/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4a07e29-b928-4427-b4c2-5315984f03b3/volumes/kubernetes.io~projected/kube-api-access-qv27q:{mountpoint:/var/lib/kubelet/pods/c4a07e29-b928-4427-b4c2-5315984f03b3/volumes/kubernetes.io~projected/kube-api-access-qv27q major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7/volumes/kubernetes.io~projected/kube-api-access-lfbgk:{mountpoint:/var/lib/kubelet/pods/c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7/volumes/kubernetes.io~projected/kube-api-access-lfbgk major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf9df1a4-3684-464c-97f3-21a3151a7183/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/cf9df1a4-3684-464c-97f3-21a3151a7183/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf9df1a4-3684-464c-97f3-21a3151a7183/volumes/kubernetes.io~projected/kube-api-access-kgxs2:{mountpoint:/var/lib/kubelet/pods/cf9df1a4-3684-464c-97f3-21a3151a7183/volumes/kubernetes.io~projected/kube-api-access-kgxs2 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf9df1a4-3684-464c-97f3-21a3151a7183/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/cf9df1a4-3684-464c-97f3-21a3151a7183/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf9df1a4-3684-464c-97f3-21a3151a7183/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/cf9df1a4-3684-464c-97f3-21a3151a7183/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3acf985-6085-4061-8ca8-e6d7036cb6fd/volumes/kubernetes.io~projected/kube-api-access-x5sx2:{mountpoint:/var/lib/kubelet/pods/d3acf985-6085-4061-8ca8-e6d7036cb6fd/volumes/kubernetes.io~projected/kube-api-access-x5sx2 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d45d9164-2ba4-400d-8b27-273d7f4f351d/volumes/kubernetes.io~projected/kube-api-access-txdkn:{mountpoint:/var/lib/kubelet/pods/d45d9164-2ba4-400d-8b27-273d7f4f351d/volumes/kubernetes.io~projected/kube-api-access-txdkn major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d49ad8b8-08aa-4da0-9139-b2be1efeb6cd/volumes/kubernetes.io~projected/kube-api-access-gqv42:{mountpoint:/var/lib/kubelet/pods/d49ad8b8-08aa-4da0-9139-b2be1efeb6cd/volumes/kubernetes.io~projected/kube-api-access-gqv42 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d992040e-0d1e-4227-befb-cfef1acd8aad/volumes/kubernetes.io~projected/kube-api-access-rchb2:{mountpoint:/var/lib/kubelet/pods/d992040e-0d1e-4227-befb-cfef1acd8aad/volumes/kubernetes.io~projected/kube-api-access-rchb2 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f264bf41-80c4-4dda-a67e-6651b464655c/volumes/kubernetes.io~projected/kube-api-access-gw9lp:{mountpoint:/var/lib/kubelet/pods/f264bf41-80c4-4dda-a67e-6651b464655c/volumes/kubernetes.io~projected/kube-api-access-gw9lp major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f264bf41-80c4-4dda-a67e-6651b464655c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f264bf41-80c4-4dda-a67e-6651b464655c/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fecc158a-a288-4064-b3e9-6fe91e703f5b/volumes/kubernetes.io~projected/kube-api-access-7w7df:{mountpoint:/var/lib/kubelet/pods/fecc158a-a288-4064-b3e9-6fe91e703f5b/volumes/kubernetes.io~projected/kube-api-access-7w7df major:0 minor:227 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/5ae5591025c60976dc1f77c4b267401bf4e2e49e8040e5e500dc290c8868406c/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/269512a427d086d5d26aa0671c7694034d999e77cbc51a9d9ea6518068e34124/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/598bf1cbfdd661c736b9bf63b565cf0ab62d0264784ad1ba33ffa0c482127489/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/1dbfda814c6f1913df4fe984fcaa3daae4ef927c4d72550ecedcd6913a348afa/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-112:{mountpoint:/var/lib/containers/storage/overlay/cf83f145b00b074ce16ccbdc253d2cd138f40efdeb36e803ddb851dbfd36b412/merged major:0 minor:112 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/3d3009af9e6c6a80e436474d1c868ef396e636eed7e829df5dda198a9ff7df32/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-122:{mountpoint:/var/lib/containers/storage/overlay/4ac3a9bdd8be5a52ce5a0789079f6474231a8e10858103abbc6393585d574508/merged major:0 minor:122 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/6505d4150cdf10b4e3d44c93adbcd7a8fff17f8c8a1c987a0173fe1a0841df9e/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/e576d0c77df223b1df459f9e5fb5bdc5608faccad3158cfb3ccd2b8218084186/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/c3795d0c4ec4bbbf1566ce7a621f8e63279e088731c43beda1f2360eca8663f8/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/4dd72139f175442fa6f08680543d834eada92e5df85ea0f0693468064d75dbb4/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/d1f4ea33a4ea9cb8eac186376473d8e76dd7875efc28b6ce8f4f21b360f29f71/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/75f9497aea86fa6f83d73f96df16df765e4cdb2d86fe12533102b82f04e716e6/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/00ce363be4e62e51192f924e4a8bda878df612898bc20d8ef55d417e4e8c8148/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/e68020158ff84c5b6e1c215ed6d801ea6a2a2bd4a633bede4cb06a3c3bc6a892/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/bb3fbeaae5de7ed8c12a2af7a79f2d76412b0a276682482a6494b9e1282b2126/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/036cd6a50f62090dfeea8c4ad657b9c31af4d193cebe17c0cbb83b286b920de2/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/fce2ce5d9c3993d824f0dbe4455ddbf37c6e7f3cd0fd2fa3fe76485347153241/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/e8abf6cec7f3bcd4d684c237ceca74c7986cec346b44713b1fbde8809ccd9bba/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/381bd133c35ed6945497ba2719648778a43dd502a3d203ac368ae0f98d11b57b/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/21f6bc73606e1f7523fa3c780c1c5142f4389066e836d567f760a37873ef9d78/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/d4235c1a7c49e9d929ae553879682450882dc32ed34ab701cf3af5f3884e456d/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/4d63cba65214809fc56ab57d7daeaf71e37f4c7119741ff4b4b20c0cbc3267b8/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/7953329b2a7cdd5d9b3ee6e64b9142a9fa00510afb87d7ceabbffac30c18a60a/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/6fd73556af64b96a3f9daf51cac8d63746fc9d2b749d6afea5e3a1005061c74c/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/0154550c370da93b447d0e5d82295b7e65b7f0072dcba997386820a25fea829b/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/680f614ac7bc0d3b046d7f2c96bb51d6bec56ff731eaa5c74aaed1395f353dfe/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/f727b647bd3af19ef94f4ffde80fa94ffd15751d10e8dcb2f7f09fac54eb6616/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/8a4ce8af87c6e143399badf06dfd6f1f3f259f836b08f0077ef07ba29100707b/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/60a9963507d386d898960143a08b97a8c85d62678b07d7a8be8883a188675c63/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/0d141cd27878d775a83822edb3caf71a364775806e054fe587699648dc101dfa/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/9673bfc88495e8954e2a2430eb7e7791f4fa27b0494c396846155a0c7d5d5d81/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/780c1bdf6f65607282163f5ade63ef56ff79a0d19c3c99514942474a5d5244d6/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/178c12455505ac2d11bb961bd6834a5f216ae79a1e49c4d9e004c8e42b7ddd78/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/ad38496592ec1feb8732f783adb04f58601a8134a05a60a6930eaddffcfab88e/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/9632bcf7d6b97d30590b3df417426dcb969a8063805399e5ed3936eac50bee1e/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/07405198b6a52cec6df919c64e9deb66a8798af9435ffc5e6ed7e41821645a4a/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/4181bc3fb95e2b12ab6b9c9b33ee98264c7a5185d4522096c8cde5cebec817b5/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/32135b7a42e75a81dd49f095456478d1811cf05c7693d0dfc6d988759da378bf/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/a0b0d9afd6e59d1dfbe53b5e771b479782e5f2575440054b1bee1f9c2d9e0bd8/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/4740a2d5f28ee29730b140da23b64749c8e9c53500f8d0a3259981b01f34da1f/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/82e1781ed5e59df4c2b6c4a6919d3767516f08b359c54a9e722a0ba905f27efa/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/574328e846e1b10aff803a2e8a0c44f9eb4232ad2a6fc27f9efad134d4c1ce53/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/7b4849c5394914c23c2472fbcecee947b64a51926111dfb8ff020fcaefa4d71b/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/5c9d4d3e7158134471316432552ac4675aaed8b7912e281613bc3386bf58d70d/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/07ca4d0b053f718991287e534f9c4e6c7610d3897fddd2065ff7849d3c0c7b63/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/a8b4f33715a50ff91641f5b78ec0983603b626c6834b07551a1bf349b66e09f5/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/7463c81b4014990eef16168cddf45101acbcc28bac830bf7fcd73455534691c1/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/72b5b6c35de13ee20a0ae57f2796a7570bccb0f177349c74c2d182a455d5347c/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/7311ae2e2b74ad37d6d16470466f1c78403c58e8190dc84b844c5d7c1e547abb/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/e7f27be2cc1d990f6ddcfa9c7492ee447d822ced4309d489d00a8d729a85c72b/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/0e90e92496e1b61ef35bf16b19bd64ca005c3106e1826d9ba764a0440518dfed/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/bc1297c2897e56b351eac643a79435c4cf22d75c261573581cba9ffc1ad21bb8/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/c95ec6eb0b1b80577f0156d89556b0ebcfaadf6313f72e88f8d5cd7532a41269/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/5beaa28be9111e46e821dcc4f65c7a9c724503b1729a14da57d9fe7af6164e1e/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 21:08:27.623796 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.623456 4935 manager.go:217] Machine: {Timestamp:2026-11-02 21:08:27.622962251 +0000 UTC m=+0.062536963 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100219 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23b5015d2620ed3d048a7909b84980 SystemUUID:ec23b501-5d26-20ed-3d04-8a7909b84980 BootID:a5ba59ff-65e8-4925-9405-137fc21e9914 Filesystems:[{Device:/run/containers/storage/overlay-containers/77adee5b179ff360088237e14769bbde2edd21eab5ed31dca582284317167945/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9a2820a3-6d1e-42d0-8a89-0e47f870abc4/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ebda27314aede86920f44c514e1aeb2348dcaf88ab8704a9d68111a733f1b16/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-122 DeviceMajor:0 DeviceMinor:122 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cf9df1a4-3684-464c-97f3-21a3151a7183/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6b5655ce-43ca-48e4-9e4b-872283ccb85c/volumes/kubernetes.io~projected/kube-api-access-q672k DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f264bf41-80c4-4dda-a67e-6651b464655c/volumes/kubernetes.io~projected/kube-api-access-gw9lp DeviceMajor:0 DeviceMinor:232 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8ca0bae8-7539-4b19-912c-4379064ecfa2/volumes/kubernetes.io~projected/kube-api-access-gtj26 DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d5d4f6ec3bb69138dd6f297adbe5cc65f427c366c7e7e872caa1ea7bd8e267af/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8cf65e09-2079-40cc-b597-1b67ba482a04/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7de0fc42-d787-49e7-80a2-5122a67bfba0/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/99213b29ed642ee2d9ddc00d9043868fee9f471355a4de7ce5e4cb2a7e78f446/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/42b83ba5-5e98-459f-ad4f-6a94c90212be/volumes/kubernetes.io~projected/kube-api-access-lq5t9 DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/44a209b7-2144-45a0-bea0-2f8adf6658cb/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bff8d097-26a3-4621-ac01-c3b00ed23e06/volumes/kubernetes.io~projected/kube-api-access-w6wd7 DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dc462923466524e209c628f11d6b36ff1698809fd6c547d6b1821ccb8fb05c0d/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/26a2e1e33242c28dc251c800d13b9c6c302e9c1d7705af705aea7e307fc83ec6/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/996e3208f62bddecd270a96f63112c517e921203b4ff8047fa02bb22f44c06d1/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cf9df1a4-3684-464c-97f3-21a3151a7183/volumes/kubernetes.io~projected/kube-api-access-kgxs2 DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/595bbca93c4ad043d8e398da486a2883ff60f00a9aaaaf8bcec628fde4087f03/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/7dab1353-88e8-4250-b7ab-a4b2a69e37fa/volumes/kubernetes.io~projected/kube-api-access-wgqft DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/67621105d0befe73aa74ba7e09f62ac325157669fa22233822aafe5b1fb981f2/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/bff8d097-26a3-4621-ac01-c3b00ed23e06/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ca28f2b-c77d-4aa1-8501-ec380253ab88/volumes/kubernetes.io~projected/kube-api-access-t7kkw DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8cf65e09-2079-40cc-b597-1b67ba482a04/volumes/kubernetes.io~projected/kube-api-access-5v2wr DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4ad2a39a-7a74-44a2-a97f-88e84e65616c/volumes/kubernetes.io~projected/kube-api-access-fmqww DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cf9df1a4-3684-464c-97f3-21a3151a7183/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a899c9c5-79b4-472d-942c-5e0c4c53e954/volumes/kubernetes.io~projected/kube-api-access-9gttn DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d12fd879df50acc9c58da47ef0db58f3bb45a29369edbe0f5eb5b196f9dec48/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7/volumes/kubernetes.io~projected/kube-api-access-lfbgk DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/67c3caa0-0b53-42c9-98b2-96addd3dd7fc/volumes/kubernetes.io~projected/kube-api-access-jx8dl DeviceMajor:0 DeviceMinor:316 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a7204d98-9f92-46e6-9e6d-482371e0a3ee/volumes/kubernetes.io~projected/kube-api-access-sx42g DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/be519c9c-2677-4c4a-99c5-c537317bda3e/volumes/kubernetes.io~projected/kube-api-access-fshk5 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f264bf41-80c4-4dda-a67e-6651b464655c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fecc158a-a288-4064-b3e9-6fe91e703f5b/volumes/kubernetes.io~projected/kube-api-access-7w7df DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ca28f2b-c77d-4aa1-8501-ec380253ab88/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/39830cb0-c259-446d-b2d3-fd8956b968fb/volumes/kubernetes.io~projected/kube-api-access-9kmdm DeviceMajor:0 DeviceMinor:329 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b7afbd2ef8c2fd552a7bde8d073aae109ba33a010abf7b03f5689b7c6f7c8589/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e8c8a4724dec7748c53ba6fa336407ba878e7ff4704c694f882fcce1127403de/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/43eabb098ccce10b169118e5fb17731ed32532e6e9057a0359575b38e98c8932/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/08b38b87-8fd7-475e-94db-8bdd1339dc4b/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ab0662d-ef78-415d-98f6-8b271e23e16f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-112 DeviceMajor:0 DeviceMinor:112 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4a07e29-b928-4427-b4c2-5315984f03b3/volumes/kubernetes.io~projected/kube-api-access-qv27q DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/934bd6cf-3dcb-40fd-9b11-97886b3f098b/volumes/kubernetes.io~projected/kube-api-access-vx9qw DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d189fe9831262764a041fc7b52264ccceae18308ea1905cab9e7ce1a790e5ed7/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cf9df1a4-3684-464c-97f3-21a3151a7183/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d3acf985-6085-4061-8ca8-e6d7036cb6fd/volumes/kubernetes.io~projected/kube-api-access-x5sx2 DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a47e2648877ec7c53a35a6977dc0d961c05a330311daad6d70cfe41f3b51d198/userdata/shm DeviceMajor:0 DeviceMinor:258 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a523dc598caeeb5d5889543053a3cd0656321bcf0752eb039e61ec5e9c08e5d2/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f401984caa01a6a03fd60b1964dcc5d3ff649eccb524d2222249b131590e1b8/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/44a209b7-2144-45a0-bea0-2f8adf6658cb/volumes/kubernetes.io~projected/kube-api-access-hxcnw DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a7204d98-9f92-46e6-9e6d-482371e0a3ee/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d45d9164-2ba4-400d-8b27-273d7f4f351d/volumes/kubernetes.io~projected/kube-api-access-txdkn DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9a2820a3-6d1e-42d0-8a89-0e47f870abc4/volumes/kubernetes.io~projected/kube-api-access-gnvnx DeviceMajor:0 DeviceMinor:234 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7de0fc42-d787-49e7-80a2-5122a67bfba0/volumes/kubernetes.io~projected/kube-api-access-jxm2l DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/08b38b87-8fd7-475e-94db-8bdd1339dc4b/volumes/kubernetes.io~projected/kube-api-access-bxbh5 DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed611eb381e81907dcc1e8778781305f8612bdde748f9c61c70645463827e131/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46114be710d5a4afe8f3e6fb9d6da86b2d9882a4ad235f92ef80f3710c460d13/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01c2aa740e152759c0ae026d7d14853bd1ce15949ee48d84b6c1118358ec89dd/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bff8d097-26a3-4621-ac01-c3b00ed23e06/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/345370c7-0ca8-4afe-94db-c9f5394d4d72/volumes/kubernetes.io~projected/kube-api-access-rfhs6 DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/08b38b87-8fd7-475e-94db-8bdd1339dc4b/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24cf1e198be9f0535676921dffad7301a8eec044161ab4b7507aa1fe58f1dc6d/userdata/shm DeviceMajor:0 DeviceMinor:341 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a7204d98-9f92-46e6-9e6d-482371e0a3ee/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/44a209b7-2144-45a0-bea0-2f8adf6658cb/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fea13107742b81caf647af1279a23748237c82739f6b840f2bb100c0be1127d1/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d49ad8b8-08aa-4da0-9139-b2be1efeb6cd/volumes/kubernetes.io~projected/kube-api-access-gqv42 DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d992040e-0d1e-4227-befb-cfef1acd8aad/volumes/kubernetes.io~projected/kube-api-access-rchb2 DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/08b38b87-8fd7-475e-94db-8bdd1339dc4b/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/904c2e78a563c743911689439b0018c42b7c9ae7c573061d777d617010d61203/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:24cf1e198be9f05 MacAddress:fa:14:fb:94:e4:e0 Speed:10000 Mtu:8901} {Name:26a2e1e33242c28 MacAddress:32:a9:52:fe:70:9c Speed:10000 Mtu:8901} {Name:3d12fd879df50ac MacAddress:2a:e8:d5:af:50:4b Speed:10000 Mtu:8901} {Name:43eabb098ccce10 MacAddress:1e:30:6f:b1:ca:ff Speed:10000 Mtu:8901} {Name:595bbca93c4ad04 MacAddress:7a:cb:e7:30:ad:ba Speed:10000 Mtu:8901} {Name:904c2e78a563c74 MacAddress:6a:23:f9:9f:62:dc Speed:10000 Mtu:8901} {Name:99213b29ed642ee MacAddress:22:40:7d:47:19:4f Speed:10000 Mtu:8901} {Name:996e3208f62bdde MacAddress:aa:7e:4e:dd:3b:a1 Speed:10000 Mtu:8901} {Name:a47e2648877ec7c MacAddress:52:02:1d:f0:e2:c4 Speed:10000 Mtu:8901} {Name:a523dc598caeeb5 MacAddress:f2:5f:e7:d7:dc:ab Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:d6:d3:1e:85 Speed:0 Mtu:9001} {Name:br-int MacAddress:d6:76:e1:22:8a:89 Speed:0 Mtu:8901} {Name:d5d4f6ec3bb6913 MacAddress:d2:0a:fa:3b:20:8a Speed:10000 Mtu:8901} {Name:e8c8a4724dec774 MacAddress:92:fa:9e:67:5d:f6 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:d6:d3:1e:85 Speed:0 Mtu:9001} {Name:fea13107742b81c MacAddress:aa:47:e8:76:d7:3a Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:be:72:2f:9a:e2:e1 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:a6:83:24:39:e3:a3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 21:08:27.624000 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.623801 4935 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 21:08:27.624000 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.623848 4935 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 21:08:27.624224 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.624195 4935 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 21:08:27.624350 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.624225 4935 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-89.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 21:08:27.624397 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.624353 4935 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 21:08:27.624397 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.624361 4935 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 21:08:27.624397 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.624371 4935 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 21:08:27.624397 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.624387 4935 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 21:08:27.624561 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.624415 4935 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 21:08:27.624613 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.624596 4935 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 21:08:27.624707 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.624697 4935 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 21:08:27.624746 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.624732 4935 kubelet.go:493] "Attempting to sync node with API server" Nov 02 21:08:27.624746 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.624745 4935 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 21:08:27.624804 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.624756 4935 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 21:08:27.624804 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.624765 4935 kubelet.go:405] "Adding apiserver pod source" Nov 02 21:08:27.624804 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.624772 4935 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 21:08:27.625694 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.625675 4935 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 21:08:27.625694 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.625690 4935 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 21:08:27.626238 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.626223 4935 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 21:08:27.729907 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.729886 4935 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 21:08:27.729907 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.729906 4935 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 21:08:27.730048 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.730036 4935 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 21:08:27.730276 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.730260 4935 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 21:08:27.730317 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.730295 4935 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 21:08:27.730466 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.730451 4935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 21:08:27.730506 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.730477 4935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 21:08:27.730506 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.730487 4935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 21:08:27.730506 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.730493 4935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 21:08:27.730506 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.730500 4935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 21:08:27.730506 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.730505 4935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 21:08:27.730642 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.730511 4935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 21:08:27.730642 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.730517 4935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 21:08:27.730642 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.730524 4935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 21:08:27.730642 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.730530 4935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 21:08:27.730642 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.730544 4935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 21:08:27.730642 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.730554 4935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 21:08:27.730642 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.730595 4935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 21:08:27.730642 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.730601 4935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 21:08:27.734219 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.734203 4935 server.go:1263] "Started kubelet" Nov 02 21:08:27.734498 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.734427 4935 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 21:08:27.734773 ip-10-0-128-89 systemd[1]: Started Kubernetes Kubelet. Nov 02 21:08:27.735535 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.735233 4935 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 21:08:27.735627 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.735561 4935 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 21:08:27.735891 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.735870 4935 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 21:08:27.738547 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.738265 4935 server.go:317] "Adding debug handlers to kubelet server" Nov 02 21:08:27.740957 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.740933 4935 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 21:08:27.740957 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.740951 4935 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 21:08:27.742082 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.741012 4935 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:02:46 +0000 UTC" deadline="2028-04-18 01:37:11.631050134 +0000 UTC" Nov 02 21:08:27.742082 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.741031 4935 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12772h28m43.890021661s" Nov 02 21:08:27.742082 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.741046 4935 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 21:08:27.742082 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.741055 4935 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 21:08:27.742082 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:27.741072 4935 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-89.us-west-2.compute.internal\" not found" Nov 02 21:08:27.742082 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.741279 4935 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 21:08:27.743569 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.743532 4935 factory.go:153] Registering CRI-O factory Nov 02 21:08:27.743569 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.743562 4935 factory.go:223] Registration of the crio container factory successfully Nov 02 21:08:27.743737 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.743617 4935 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 21:08:27.743737 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.743625 4935 factory.go:55] Registering systemd factory Nov 02 21:08:27.743737 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.743633 4935 factory.go:223] Registration of the systemd container factory successfully Nov 02 21:08:27.743737 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.743648 4935 factory.go:103] Registering Raw factory Nov 02 21:08:27.743737 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.743660 4935 manager.go:1199] Started watching for new ooms in manager Nov 02 21:08:27.744009 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.743994 4935 manager.go:319] Starting recovery of all containers Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744583 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08b38b87-8fd7-475e-94db-8bdd1339dc4b" volumeName="kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-kube-api-access-bxbh5" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744634 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08b38b87-8fd7-475e-94db-8bdd1339dc4b" volumeName="kubernetes.io/secret/08b38b87-8fd7-475e-94db-8bdd1339dc4b-image-registry-private-configuration" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744649 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24c2ea1f-b61b-4e62-914a-2c52508bc04d" volumeName="kubernetes.io/configmap/24c2ea1f-b61b-4e62-914a-2c52508bc04d-nginx-conf" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744657 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1" volumeName="kubernetes.io/secret/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-original-pull-secret" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744664 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d992040e-0d1e-4227-befb-cfef1acd8aad" volumeName="kubernetes.io/projected/d992040e-0d1e-4227-befb-cfef1acd8aad-kube-api-access-rchb2" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744672 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ab0662d-ef78-415d-98f6-8b271e23e16f" volumeName="kubernetes.io/configmap/3ab0662d-ef78-415d-98f6-8b271e23e16f-konnectivity-ca" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744679 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7de0fc42-d787-49e7-80a2-5122a67bfba0" volumeName="kubernetes.io/projected/7de0fc42-d787-49e7-80a2-5122a67bfba0-kube-api-access-jxm2l" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744687 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cf65e09-2079-40cc-b597-1b67ba482a04" volumeName="kubernetes.io/configmap/8cf65e09-2079-40cc-b597-1b67ba482a04-service-ca-bundle" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744698 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a2820a3-6d1e-42d0-8a89-0e47f870abc4" volumeName="kubernetes.io/configmap/9a2820a3-6d1e-42d0-8a89-0e47f870abc4-trusted-ca" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744732 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7204d98-9f92-46e6-9e6d-482371e0a3ee" volumeName="kubernetes.io/projected/a7204d98-9f92-46e6-9e6d-482371e0a3ee-kube-api-access-sx42g" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744739 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f264bf41-80c4-4dda-a67e-6651b464655c" volumeName="kubernetes.io/configmap/f264bf41-80c4-4dda-a67e-6651b464655c-config" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744746 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="345370c7-0ca8-4afe-94db-c9f5394d4d72" volumeName="kubernetes.io/projected/345370c7-0ca8-4afe-94db-c9f5394d4d72-kube-api-access-rfhs6" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744754 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44a209b7-2144-45a0-bea0-2f8adf6658cb" volumeName="kubernetes.io/projected/44a209b7-2144-45a0-bea0-2f8adf6658cb-kube-api-access-hxcnw" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744762 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a899c9c5-79b4-472d-942c-5e0c4c53e954" volumeName="kubernetes.io/projected/a899c9c5-79b4-472d-942c-5e0c4c53e954-kube-api-access-9gttn" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744771 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bff8d097-26a3-4621-ac01-c3b00ed23e06" volumeName="kubernetes.io/projected/bff8d097-26a3-4621-ac01-c3b00ed23e06-kube-api-access-w6wd7" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744779 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08b38b87-8fd7-475e-94db-8bdd1339dc4b" volumeName="kubernetes.io/configmap/08b38b87-8fd7-475e-94db-8bdd1339dc4b-trusted-ca" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744805 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08b38b87-8fd7-475e-94db-8bdd1339dc4b" volumeName="kubernetes.io/secret/08b38b87-8fd7-475e-94db-8bdd1339dc4b-installation-pull-secrets" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744813 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ab0662d-ef78-415d-98f6-8b271e23e16f" volumeName="kubernetes.io/secret/3ab0662d-ef78-415d-98f6-8b271e23e16f-agent-certs" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744820 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ca0bae8-7539-4b19-912c-4379064ecfa2" volumeName="kubernetes.io/configmap/8ca0bae8-7539-4b19-912c-4379064ecfa2-multus-daemon-config" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744828 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4a07e29-b928-4427-b4c2-5315984f03b3" volumeName="kubernetes.io/projected/c4a07e29-b928-4427-b4c2-5315984f03b3-kube-api-access-qv27q" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744835 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08b38b87-8fd7-475e-94db-8bdd1339dc4b" volumeName="kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-bound-sa-token" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744843 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b5655ce-43ca-48e4-9e4b-872283ccb85c" volumeName="kubernetes.io/configmap/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-config" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744851 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dab1353-88e8-4250-b7ab-a4b2a69e37fa" volumeName="kubernetes.io/configmap/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744858 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cf65e09-2079-40cc-b597-1b67ba482a04" volumeName="kubernetes.io/empty-dir/8cf65e09-2079-40cc-b597-1b67ba482a04-tmp" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744865 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7204d98-9f92-46e6-9e6d-482371e0a3ee" volumeName="kubernetes.io/empty-dir/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-tuned" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744873 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bff8d097-26a3-4621-ac01-c3b00ed23e06" volumeName="kubernetes.io/configmap/bff8d097-26a3-4621-ac01-c3b00ed23e06-env-overrides" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744881 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7" volumeName="kubernetes.io/projected/c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7-kube-api-access-lfbgk" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744888 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d45d9164-2ba4-400d-8b27-273d7f4f351d" volumeName="kubernetes.io/projected/d45d9164-2ba4-400d-8b27-273d7f4f351d-kube-api-access-txdkn" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744901 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ad2a39a-7a74-44a2-a97f-88e84e65616c" volumeName="kubernetes.io/configmap/4ad2a39a-7a74-44a2-a97f-88e84e65616c-config-volume" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744909 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ca28f2b-c77d-4aa1-8501-ec380253ab88" volumeName="kubernetes.io/secret/5ca28f2b-c77d-4aa1-8501-ec380253ab88-serving-cert" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744916 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67c3caa0-0b53-42c9-98b2-96addd3dd7fc" volumeName="kubernetes.io/projected/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-kube-api-access-jx8dl" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744924 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cf65e09-2079-40cc-b597-1b67ba482a04" volumeName="kubernetes.io/projected/8cf65e09-2079-40cc-b597-1b67ba482a04-kube-api-access-5v2wr" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744944 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf9df1a4-3684-464c-97f3-21a3151a7183" volumeName="kubernetes.io/configmap/cf9df1a4-3684-464c-97f3-21a3151a7183-trusted-ca" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744953 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf9df1a4-3684-464c-97f3-21a3151a7183" volumeName="kubernetes.io/empty-dir/cf9df1a4-3684-464c-97f3-21a3151a7183-ca-trust-extracted" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744961 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3acf985-6085-4061-8ca8-e6d7036cb6fd" volumeName="kubernetes.io/projected/d3acf985-6085-4061-8ca8-e6d7036cb6fd-kube-api-access-x5sx2" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744967 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d49ad8b8-08aa-4da0-9139-b2be1efeb6cd" volumeName="kubernetes.io/configmap/d49ad8b8-08aa-4da0-9139-b2be1efeb6cd-serviceca" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744976 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08b38b87-8fd7-475e-94db-8bdd1339dc4b" volumeName="kubernetes.io/empty-dir/08b38b87-8fd7-475e-94db-8bdd1339dc4b-ca-trust-extracted" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744985 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44a209b7-2144-45a0-bea0-2f8adf6658cb" volumeName="kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-default-certificate" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744992 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dab1353-88e8-4250-b7ab-a4b2a69e37fa" volumeName="kubernetes.io/projected/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-kube-api-access-wgqft" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.744999 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ca0bae8-7539-4b19-912c-4379064ecfa2" volumeName="kubernetes.io/projected/8ca0bae8-7539-4b19-912c-4379064ecfa2-kube-api-access-gtj26" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745006 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cf65e09-2079-40cc-b597-1b67ba482a04" volumeName="kubernetes.io/configmap/8cf65e09-2079-40cc-b597-1b67ba482a04-trusted-ca-bundle" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745013 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d992040e-0d1e-4227-befb-cfef1acd8aad" volumeName="kubernetes.io/configmap/d992040e-0d1e-4227-befb-cfef1acd8aad-iptables-alerter-script" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745020 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39830cb0-c259-446d-b2d3-fd8956b968fb" volumeName="kubernetes.io/projected/39830cb0-c259-446d-b2d3-fd8956b968fb-kube-api-access-9kmdm" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745027 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42b83ba5-5e98-459f-ad4f-6a94c90212be" volumeName="kubernetes.io/projected/42b83ba5-5e98-459f-ad4f-6a94c90212be-kube-api-access-lq5t9" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745035 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44a209b7-2144-45a0-bea0-2f8adf6658cb" volumeName="kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-stats-auth" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745041 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b5655ce-43ca-48e4-9e4b-872283ccb85c" volumeName="kubernetes.io/projected/6b5655ce-43ca-48e4-9e4b-872283ccb85c-kube-api-access-q672k" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745049 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7de0fc42-d787-49e7-80a2-5122a67bfba0" volumeName="kubernetes.io/secret/7de0fc42-d787-49e7-80a2-5122a67bfba0-cert" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745056 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7204d98-9f92-46e6-9e6d-482371e0a3ee" volumeName="kubernetes.io/empty-dir/a7204d98-9f92-46e6-9e6d-482371e0a3ee-tmp" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745064 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be519c9c-2677-4c4a-99c5-c537317bda3e" volumeName="kubernetes.io/empty-dir/be519c9c-2677-4c4a-99c5-c537317bda3e-tmp-dir" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745073 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be519c9c-2677-4c4a-99c5-c537317bda3e" volumeName="kubernetes.io/projected/be519c9c-2677-4c4a-99c5-c537317bda3e-kube-api-access-fshk5" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745080 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ad2a39a-7a74-44a2-a97f-88e84e65616c" volumeName="kubernetes.io/empty-dir/4ad2a39a-7a74-44a2-a97f-88e84e65616c-tmp-dir" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745090 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ca28f2b-c77d-4aa1-8501-ec380253ab88" volumeName="kubernetes.io/configmap/5ca28f2b-c77d-4aa1-8501-ec380253ab88-config" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745099 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="934bd6cf-3dcb-40fd-9b11-97886b3f098b" volumeName="kubernetes.io/projected/934bd6cf-3dcb-40fd-9b11-97886b3f098b-kube-api-access-vx9qw" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745107 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a2820a3-6d1e-42d0-8a89-0e47f870abc4" volumeName="kubernetes.io/secret/9a2820a3-6d1e-42d0-8a89-0e47f870abc4-serving-cert" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745116 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf9df1a4-3684-464c-97f3-21a3151a7183" volumeName="kubernetes.io/secret/cf9df1a4-3684-464c-97f3-21a3151a7183-image-registry-private-configuration" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745124 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f264bf41-80c4-4dda-a67e-6651b464655c" volumeName="kubernetes.io/secret/f264bf41-80c4-4dda-a67e-6651b464655c-serving-cert" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745174 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67c3caa0-0b53-42c9-98b2-96addd3dd7fc" volumeName="kubernetes.io/empty-dir/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-data-volume" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745181 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cf65e09-2079-40cc-b597-1b67ba482a04" volumeName="kubernetes.io/secret/8cf65e09-2079-40cc-b597-1b67ba482a04-serving-cert" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745189 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bff8d097-26a3-4621-ac01-c3b00ed23e06" volumeName="kubernetes.io/configmap/bff8d097-26a3-4621-ac01-c3b00ed23e06-ovnkube-script-lib" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745197 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf9df1a4-3684-464c-97f3-21a3151a7183" volumeName="kubernetes.io/configmap/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-certificates" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745204 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d49ad8b8-08aa-4da0-9139-b2be1efeb6cd" volumeName="kubernetes.io/projected/d49ad8b8-08aa-4da0-9139-b2be1efeb6cd-kube-api-access-gqv42" seLinuxMountContext="" Nov 02 21:08:27.745161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745213 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a899c9c5-79b4-472d-942c-5e0c4c53e954" volumeName="kubernetes.io/configmap/a899c9c5-79b4-472d-942c-5e0c4c53e954-telemetry-config" seLinuxMountContext="" Nov 02 21:08:27.747611 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745220 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dab1353-88e8-4250-b7ab-a4b2a69e37fa" volumeName="kubernetes.io/configmap/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-cni-binary-copy" seLinuxMountContext="" Nov 02 21:08:27.747611 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745228 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ca28f2b-c77d-4aa1-8501-ec380253ab88" volumeName="kubernetes.io/projected/5ca28f2b-c77d-4aa1-8501-ec380253ab88-kube-api-access-t7kkw" seLinuxMountContext="" Nov 02 21:08:27.747611 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745236 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a2820a3-6d1e-42d0-8a89-0e47f870abc4" volumeName="kubernetes.io/projected/9a2820a3-6d1e-42d0-8a89-0e47f870abc4-kube-api-access-gnvnx" seLinuxMountContext="" Nov 02 21:08:27.747611 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745244 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dab1353-88e8-4250-b7ab-a4b2a69e37fa" volumeName="kubernetes.io/configmap/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 21:08:27.747611 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745251 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cf65e09-2079-40cc-b597-1b67ba482a04" volumeName="kubernetes.io/empty-dir/8cf65e09-2079-40cc-b597-1b67ba482a04-snapshots" seLinuxMountContext="" Nov 02 21:08:27.747611 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745260 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf9df1a4-3684-464c-97f3-21a3151a7183" volumeName="kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-bound-sa-token" seLinuxMountContext="" Nov 02 21:08:27.747611 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745268 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f264bf41-80c4-4dda-a67e-6651b464655c" volumeName="kubernetes.io/projected/f264bf41-80c4-4dda-a67e-6651b464655c-kube-api-access-gw9lp" seLinuxMountContext="" Nov 02 21:08:27.747611 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745275 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08b38b87-8fd7-475e-94db-8bdd1339dc4b" volumeName="kubernetes.io/configmap/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-certificates" seLinuxMountContext="" Nov 02 21:08:27.747611 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745282 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bff8d097-26a3-4621-ac01-c3b00ed23e06" volumeName="kubernetes.io/configmap/bff8d097-26a3-4621-ac01-c3b00ed23e06-ovnkube-config" seLinuxMountContext="" Nov 02 21:08:27.747611 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745289 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf9df1a4-3684-464c-97f3-21a3151a7183" volumeName="kubernetes.io/secret/cf9df1a4-3684-464c-97f3-21a3151a7183-installation-pull-secrets" seLinuxMountContext="" Nov 02 21:08:27.747611 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745296 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ad2a39a-7a74-44a2-a97f-88e84e65616c" volumeName="kubernetes.io/projected/4ad2a39a-7a74-44a2-a97f-88e84e65616c-kube-api-access-fmqww" seLinuxMountContext="" Nov 02 21:08:27.747611 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745304 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a2820a3-6d1e-42d0-8a89-0e47f870abc4" volumeName="kubernetes.io/configmap/9a2820a3-6d1e-42d0-8a89-0e47f870abc4-config" seLinuxMountContext="" Nov 02 21:08:27.747611 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745311 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf9df1a4-3684-464c-97f3-21a3151a7183" volumeName="kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-kube-api-access-kgxs2" seLinuxMountContext="" Nov 02 21:08:27.747611 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745318 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ca0bae8-7539-4b19-912c-4379064ecfa2" volumeName="kubernetes.io/configmap/8ca0bae8-7539-4b19-912c-4379064ecfa2-cni-binary-copy" seLinuxMountContext="" Nov 02 21:08:27.747611 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745325 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bff8d097-26a3-4621-ac01-c3b00ed23e06" volumeName="kubernetes.io/secret/bff8d097-26a3-4621-ac01-c3b00ed23e06-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 21:08:27.747611 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745332 4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fecc158a-a288-4064-b3e9-6fe91e703f5b" volumeName="kubernetes.io/projected/fecc158a-a288-4064-b3e9-6fe91e703f5b-kube-api-access-7w7df" seLinuxMountContext="" Nov 02 21:08:27.747611 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745339 4935 reconstruct.go:97] "Volume reconstruction finished" Nov 02 21:08:27.747611 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.745345 4935 reconciler.go:29] "Reconciler: start to sync state" Nov 02 21:08:27.747611 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.746443 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/1.log" Nov 02 21:08:27.751097 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.751065 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/1.log" Nov 02 21:08:27.759668 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.759634 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 21:08:27.764508 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.764485 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 21:08:27.767304 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.767273 4935 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 21:08:27.768615 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.768591 4935 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 21:08:27.768615 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.768616 4935 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 21:08:27.768762 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.768637 4935 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 21:08:27.768762 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:27.768678 4935 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 21:08:27.781274 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.781246 4935 generic.go:356] "Generic (PLEG): container finished" podID="7dab1353-88e8-4250-b7ab-a4b2a69e37fa" containerID="f1fae03813ba6dabd2a6fd55d4c5331ec3dd74d39091576e17bc1489c5ec7259" exitCode=0 Nov 02 21:08:27.781274 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.781272 4935 generic.go:356] "Generic (PLEG): container finished" podID="7dab1353-88e8-4250-b7ab-a4b2a69e37fa" containerID="df1cde4b7b09523fc50163615d6b83908fb0bc65988da58df5899ec5de7c311a" exitCode=0 Nov 02 21:08:27.781274 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.781279 4935 generic.go:356] "Generic (PLEG): container finished" podID="7dab1353-88e8-4250-b7ab-a4b2a69e37fa" containerID="eb2a444d2a339d2a1c6a991b86c8ace00f8093a01d10dca33f604c755ccaaae7" exitCode=0 Nov 02 21:08:27.781461 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.781285 4935 generic.go:356] "Generic (PLEG): container finished" podID="7dab1353-88e8-4250-b7ab-a4b2a69e37fa" containerID="729d877e2b17c4de601d4f03d6d1ae62895b3d7e6990325390e987bd77f855a5" exitCode=0 Nov 02 21:08:27.781461 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.781291 4935 generic.go:356] "Generic (PLEG): container finished" podID="7dab1353-88e8-4250-b7ab-a4b2a69e37fa" containerID="cbe28e728a99a4cdaf6d48995183b5e93e447192107c156b3e83a3d177f8615e" exitCode=0 Nov 02 21:08:27.781461 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.781296 4935 generic.go:356] "Generic (PLEG): container finished" podID="7dab1353-88e8-4250-b7ab-a4b2a69e37fa" containerID="a3b923e4c2fa18f9821bd7e6344c8f9147e38b3f84e726b9fa40cf8345ab57ba" exitCode=0 Nov 02 21:08:27.801561 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.801537 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 21:08:27.801942 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.801908 4935 generic.go:356] "Generic (PLEG): container finished" podID="bff8d097-26a3-4621-ac01-c3b00ed23e06" containerID="c6cb2944006d1ee69c92ed1bb854345a65049bfff5f2cf4a817992c7eb338b8e" exitCode=1 Nov 02 21:08:27.803547 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.803529 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/1.log" Nov 02 21:08:27.803655 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.803556 4935 generic.go:356] "Generic (PLEG): container finished" podID="9a2820a3-6d1e-42d0-8a89-0e47f870abc4" containerID="25880bea858c09ebae11d9d77d74a3d9d1f8c364288f4a9ac3048abab9b30755" exitCode=255 Nov 02 21:08:27.815667 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.815630 4935 generic.go:356] "Generic (PLEG): container finished" podID="30622891ad10d753b5b3452735f59810" containerID="39b9b34b579ceea3f81bca21394f7c2208287cfd9eee929286d43c55f6c4818e" exitCode=0 Nov 02 21:08:27.822667 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.822644 4935 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:08:27.834645 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.834624 4935 manager.go:324] Recovery completed Nov 02 21:08:27.856817 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.856794 4935 cpu_manager.go:226] "Starting" policy="none" Nov 02 21:08:27.856817 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.856813 4935 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 21:08:27.856999 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.856834 4935 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 21:08:27.856999 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.856983 4935 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 21:08:27.857056 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.856997 4935 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 21:08:27.857056 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.857022 4935 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 21:08:27.857056 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.857029 4935 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 21:08:27.857056 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.857036 4935 policy_none.go:50] "Start" Nov 02 21:08:27.857187 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.857059 4935 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 21:08:27.857187 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.857075 4935 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 21:08:27.857247 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.857228 4935 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 21:08:27.857247 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.857235 4935 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 21:08:27.857247 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.857241 4935 policy_none.go:44] "Start" Nov 02 21:08:27.861509 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.861478 4935 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 21:08:27.864088 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.864073 4935 manager.go:341] "Starting Device Plugin manager" Nov 02 21:08:27.864174 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:27.864104 4935 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 21:08:27.864174 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.864116 4935 server.go:87] "Starting device plugin registration server" Nov 02 21:08:27.864435 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.864421 4935 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 21:08:27.864495 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.864436 4935 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 21:08:27.864542 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.864509 4935 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 21:08:27.864654 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.864621 4935 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 21:08:27.864722 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.864667 4935 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 21:08:27.864722 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.864675 4935 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 21:08:27.865562 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.864770 4935 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T21:08:27.864652295Z","UUID":"d5570c9c-8da3-450e-b2f6-d20b0f1b5c2e","Handler":null,"Name":"","Endpoint":""} Nov 02 21:08:27.865963 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:27.865942 4935 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 21:08:27.866974 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.866788 4935 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 21:08:27.866974 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.866828 4935 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 21:08:27.868852 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.868813 4935 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-89.us-west-2.compute.internal"] Nov 02 21:08:27.869689 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.869650 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-89.us-west-2.compute.internal" event={"ID":"398d07e61029ddf8dea61a5829612cad","Type":"ContainerStarted","Data":"532f778c076c1a2b47d6ad924e142963e3d95997f7e5a7b4d7dcafffe92ac96f"} Nov 02 21:08:27.869748 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.869701 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-89.us-west-2.compute.internal" event={"ID":"398d07e61029ddf8dea61a5829612cad","Type":"ContainerStarted","Data":"ed611eb381e81907dcc1e8778781305f8612bdde748f9c61c70645463827e131"} Nov 02 21:08:27.869807 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.869794 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal" event={"ID":"30622891ad10d753b5b3452735f59810","Type":"ContainerStarted","Data":"0e96a624cbb371d3b047de96fca96c163d686d7008f4a1284be47bb01eed6b18"} Nov 02 21:08:27.869839 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.869817 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal" event={"ID":"30622891ad10d753b5b3452735f59810","Type":"ContainerDied","Data":"39b9b34b579ceea3f81bca21394f7c2208287cfd9eee929286d43c55f6c4818e"} Nov 02 21:08:27.869867 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.869836 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal" event={"ID":"30622891ad10d753b5b3452735f59810","Type":"ContainerStarted","Data":"0ebda27314aede86920f44c514e1aeb2348dcaf88ab8704a9d68111a733f1b16"} Nov 02 21:08:27.870092 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.870079 4935 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:08:27.870694 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.870459 4935 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:08:27.871675 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.871659 4935 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:08:27.871756 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.871739 4935 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:08:27.872308 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.872287 4935 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 21:08:27.872406 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.872251 4935 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:08:27.931311 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.931251 4935 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:08:27.931311 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:27.931313 4935 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-89.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:08:27.931989 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.931965 4935 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:08:27.932100 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:27.932005 4935 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:08:27.965282 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.965244 4935 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:08:27.966667 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.966641 4935 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-89.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:08:27.966781 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.966687 4935 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-89.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:08:27.966781 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.966703 4935 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-89.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:08:27.966781 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.966761 4935 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:08:27.972543 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.972514 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/398d07e61029ddf8dea61a5829612cad-config\") pod \"kube-apiserver-proxy-ip-10-0-128-89.us-west-2.compute.internal\" (UID: \"398d07e61029ddf8dea61a5829612cad\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:08:27.972662 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.972546 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/30622891ad10d753b5b3452735f59810-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal\" (UID: \"30622891ad10d753b5b3452735f59810\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:08:27.972662 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:27.972564 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30622891ad10d753b5b3452735f59810-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal\" (UID: \"30622891ad10d753b5b3452735f59810\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:08:28.069723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.069691 4935 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:08:28.069889 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.069775 4935 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:08:28.073540 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.073518 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/398d07e61029ddf8dea61a5829612cad-config\") pod \"kube-apiserver-proxy-ip-10-0-128-89.us-west-2.compute.internal\" (UID: \"398d07e61029ddf8dea61a5829612cad\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:08:28.073651 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.073544 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/30622891ad10d753b5b3452735f59810-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal\" (UID: \"30622891ad10d753b5b3452735f59810\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:08:28.073651 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.073562 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30622891ad10d753b5b3452735f59810-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal\" (UID: \"30622891ad10d753b5b3452735f59810\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:08:28.073651 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.073611 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30622891ad10d753b5b3452735f59810-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal\" (UID: \"30622891ad10d753b5b3452735f59810\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:08:28.073651 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.073618 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/398d07e61029ddf8dea61a5829612cad-config\") pod \"kube-apiserver-proxy-ip-10-0-128-89.us-west-2.compute.internal\" (UID: \"398d07e61029ddf8dea61a5829612cad\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:08:28.073651 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.073625 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/30622891ad10d753b5b3452735f59810-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal\" (UID: \"30622891ad10d753b5b3452735f59810\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal" Nov 02 21:08:28.625047 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.625011 4935 apiserver.go:52] "Watching apiserver" Nov 02 21:08:28.814944 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.814910 4935 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 21:08:28.815876 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.815842 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nv2sw","openshift-dns/node-resolver-gxdgr","openshift-console-operator/console-operator-7cd56b8dbc-qd9n6","openshift-network-operator/iptables-alerter-zpkht","openshift-cluster-node-tuning-operator/tuned-z5hfn","openshift-image-registry/node-ca-77xn9","openshift-image-registry/image-registry-588654849f-w89ht","openshift-image-registry/image-registry-c987dc54-2p2h6","openshift-insights/insights-operator-7c4986d449-2ftg6","openshift-network-console/networking-console-plugin-86c478bb88-qpklq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.us-west-2.compute.internal","openshift-service-ca-operator/service-ca-operator-544d749c8c-pfjlk","openshift-multus/multus-6blt8","openshift-multus/multus-additional-cni-plugins-94zch","openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6","kube-system/kas-connection-checker-685895c7c9-5zc9z","openshift-multus/network-metrics-daemon-s98xl","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qpf24","kube-system/konnectivity-agent-mz5qm","openshift-dns/dns-default-vn62q","kube-system/kube-apiserver-proxy-ip-10-0-128-89.us-west-2.compute.internal","openshift-ingress-canary/ingress-canary-mlgzw","kube-system/global-pull-secret-syncer-bllxh","openshift-kube-storage-version-migrator/migrator-66df5d496b-wf84d","openshift-network-diagnostics/network-check-source-6f78d59f99-8jclh","openshift-network-diagnostics/network-check-target-8c67p","openshift-insights/insights-runtime-extractor-2nshl","openshift-ovn-kubernetes/ovnkube-node-pqnmg","kube-system/kas-connection-checker-685895c7c9-bb7st","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq","openshift-ingress/router-default-798b7b6b45-hlcl7"] Nov 02 21:08:28.817920 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.817894 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" Nov 02 21:08:28.818623 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.818596 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:28.823116 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.823093 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6" Nov 02 21:08:28.824020 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.823996 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:28.824839 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.824821 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-qpklq" Nov 02 21:08:28.826220 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.826200 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:08:28.826665 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.826642 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:28.832406 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.832386 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:28.833023 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.833007 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:28.843577 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.843555 4935 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 21:08:28.866650 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.866627 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 02 21:08:28.866902 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.866885 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 02 21:08:28.866950 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.866901 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-2pl68\"" Nov 02 21:08:28.866950 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.866912 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-5jdjp\"" Nov 02 21:08:28.867388 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.867370 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 21:08:28.867458 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.867406 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qshj9\"" Nov 02 21:08:28.877619 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.877574 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cf9df1a4-3684-464c-97f3-21a3151a7183-ca-trust-extracted\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:28.877619 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.877600 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:28.877619 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.877616 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-dbus\") pod \"global-pull-secret-syncer-bllxh\" (UID: \"87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1\") " pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:08:28.877746 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.877633 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ab0662d-ef78-415d-98f6-8b271e23e16f-konnectivity-ca\") pod \"konnectivity-agent-mz5qm\" (UID: \"3ab0662d-ef78-415d-98f6-8b271e23e16f\") " pod="kube-system/konnectivity-agent-mz5qm" Nov 02 21:08:28.877746 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.877649 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx42g\" (UniqueName: \"kubernetes.io/projected/a7204d98-9f92-46e6-9e6d-482371e0a3ee-kube-api-access-sx42g\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.877746 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.877710 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-z4kx6\" (UID: \"a899c9c5-79b4-472d-942c-5e0c4c53e954\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6" Nov 02 21:08:28.877897 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.877753 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d49ad8b8-08aa-4da0-9139-b2be1efeb6cd-host\") pod \"node-ca-77xn9\" (UID: \"d49ad8b8-08aa-4da0-9139-b2be1efeb6cd\") " pod="openshift-image-registry/node-ca-77xn9" Nov 02 21:08:28.877897 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.877782 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8ca0bae8-7539-4b19-912c-4379064ecfa2-multus-daemon-config\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.877897 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.877843 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-stats-auth\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:28.877897 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.877867 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-log-socket\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.877897 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.877887 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q672k\" (UniqueName: \"kubernetes.io/projected/6b5655ce-43ca-48e4-9e4b-872283ccb85c-kube-api-access-q672k\") pod \"cluster-samples-operator-6ddb6c8b94-vmqkq\" (UID: \"6b5655ce-43ca-48e4-9e4b-872283ccb85c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" Nov 02 21:08:28.878057 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.877902 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cf9df1a4-3684-464c-97f3-21a3151a7183-ca-trust-extracted\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:28.878057 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.877913 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kmdm\" (UniqueName: \"kubernetes.io/projected/39830cb0-c259-446d-b2d3-fd8956b968fb-kube-api-access-9kmdm\") pod \"migrator-66df5d496b-wf84d\" (UID: \"39830cb0-c259-446d-b2d3-fd8956b968fb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wf84d" Nov 02 21:08:28.878057 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.877934 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxbh5\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-kube-api-access-bxbh5\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:28.878211 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.878066 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-run-ovn\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.878458 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.878099 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:08:28.878570 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.878477 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq5t9\" (UniqueName: \"kubernetes.io/projected/42b83ba5-5e98-459f-ad4f-6a94c90212be-kube-api-access-lq5t9\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:08:28.878570 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.878519 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7de0fc42-d787-49e7-80a2-5122a67bfba0-cert\") pod \"ingress-canary-mlgzw\" (UID: \"7de0fc42-d787-49e7-80a2-5122a67bfba0\") " pod="openshift-ingress-canary/ingress-canary-mlgzw" Nov 02 21:08:28.878662 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.878562 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ca0bae8-7539-4b19-912c-4379064ecfa2-cni-binary-copy\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.878662 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.878644 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be519c9c-2677-4c4a-99c5-c537317bda3e-tmp-dir\") pod \"node-resolver-gxdgr\" (UID: \"be519c9c-2677-4c4a-99c5-c537317bda3e\") " pod="openshift-dns/node-resolver-gxdgr" Nov 02 21:08:28.878758 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.878686 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qpklq\" (UID: \"24c2ea1f-b61b-4e62-914a-2c52508bc04d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qpklq" Nov 02 21:08:28.878758 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.878731 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-system-cni-dir\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:08:28.878847 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.878770 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-hostroot\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.878847 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.878827 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w7df\" (UniqueName: \"kubernetes.io/projected/fecc158a-a288-4064-b3e9-6fe91e703f5b-kube-api-access-7w7df\") pod \"network-check-source-6f78d59f99-8jclh\" (UID: \"fecc158a-a288-4064-b3e9-6fe91e703f5b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8jclh" Nov 02 21:08:28.878935 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.878871 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be519c9c-2677-4c4a-99c5-c537317bda3e-tmp-dir\") pod \"node-resolver-gxdgr\" (UID: \"be519c9c-2677-4c4a-99c5-c537317bda3e\") " pod="openshift-dns/node-resolver-gxdgr" Nov 02 21:08:28.878979 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.878931 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cf65e09-2079-40cc-b597-1b67ba482a04-service-ca-bundle\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:28.879038 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.879021 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqv42\" (UniqueName: \"kubernetes.io/projected/d49ad8b8-08aa-4da0-9139-b2be1efeb6cd-kube-api-access-gqv42\") pod \"node-ca-77xn9\" (UID: \"d49ad8b8-08aa-4da0-9139-b2be1efeb6cd\") " pod="openshift-image-registry/node-ca-77xn9" Nov 02 21:08:28.879085 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.879065 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gttn\" (UniqueName: \"kubernetes.io/projected/a899c9c5-79b4-472d-942c-5e0c4c53e954-kube-api-access-9gttn\") pod \"cluster-monitoring-operator-884576c57-z4kx6\" (UID: \"a899c9c5-79b4-472d-942c-5e0c4c53e954\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6" Nov 02 21:08:28.879151 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.879088 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-var-lib-kubelet\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.879214 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.879200 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cf9df1a4-3684-464c-97f3-21a3151a7183-installation-pull-secrets\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:28.879262 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.879237 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-bound-sa-token\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:28.879308 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.879284 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8cf65e09-2079-40cc-b597-1b67ba482a04-serving-cert\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:28.879354 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.879313 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:08:28.879401 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.879354 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfhs6\" (UniqueName: \"kubernetes.io/projected/345370c7-0ca8-4afe-94db-c9f5394d4d72-kube-api-access-rfhs6\") pod \"kas-connection-checker-685895c7c9-nv2sw\" (UID: \"345370c7-0ca8-4afe-94db-c9f5394d4d72\") " pod="kube-system/kas-connection-checker-685895c7c9-nv2sw" Nov 02 21:08:28.879401 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.879389 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-certificates\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:28.879572 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.879548 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8cf65e09-2079-40cc-b597-1b67ba482a04-serving-cert\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:28.879572 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.879564 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cf9df1a4-3684-464c-97f3-21a3151a7183-trusted-ca\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:28.879676 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.879552 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cf9df1a4-3684-464c-97f3-21a3151a7183-installation-pull-secrets\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:28.879676 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.879602 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:28.879676 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.879642 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-kubernetes\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.879810 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.879684 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgqft\" (UniqueName: \"kubernetes.io/projected/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-kube-api-access-wgqft\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:08:28.879861 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.879816 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f264bf41-80c4-4dda-a67e-6651b464655c-config\") pod \"service-ca-operator-544d749c8c-pfjlk\" (UID: \"f264bf41-80c4-4dda-a67e-6651b464655c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pfjlk" Nov 02 21:08:28.879906 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.879878 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw9lp\" (UniqueName: \"kubernetes.io/projected/f264bf41-80c4-4dda-a67e-6651b464655c-kube-api-access-gw9lp\") pod \"service-ca-operator-544d749c8c-pfjlk\" (UID: \"f264bf41-80c4-4dda-a67e-6651b464655c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pfjlk" Nov 02 21:08:28.880141 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.879941 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ca28f2b-c77d-4aa1-8501-ec380253ab88-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf\" (UID: \"5ca28f2b-c77d-4aa1-8501-ec380253ab88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf" Nov 02 21:08:28.880227 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.880173 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9a2820a3-6d1e-42d0-8a89-0e47f870abc4-serving-cert\") pod \"console-operator-7cd56b8dbc-qd9n6\" (UID: \"9a2820a3-6d1e-42d0-8a89-0e47f870abc4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:28.880280 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.880226 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08b38b87-8fd7-475e-94db-8bdd1339dc4b-ca-trust-extracted\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:28.880280 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.880269 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx9qw\" (UniqueName: \"kubernetes.io/projected/934bd6cf-3dcb-40fd-9b11-97886b3f098b-kube-api-access-vx9qw\") pod \"volume-data-source-validator-58c9f5f46d-qpf24\" (UID: \"934bd6cf-3dcb-40fd-9b11-97886b3f098b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qpf24" Nov 02 21:08:28.880370 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.880353 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:28.880421 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.880400 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:08:28.880471 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.880444 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-bound-sa-token\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:28.880551 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.880535 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-run-ovn-kubernetes\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.880599 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.880547 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08b38b87-8fd7-475e-94db-8bdd1339dc4b-ca-trust-extracted\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:28.880599 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.880574 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bff8d097-26a3-4621-ac01-c3b00ed23e06-ovnkube-config\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.880694 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.880602 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfbgk\" (UniqueName: \"kubernetes.io/projected/c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7-kube-api-access-lfbgk\") pod \"network-check-target-8c67p\" (UID: \"c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7\") " pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:08:28.880694 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.880634 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-node-log\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.880694 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.880667 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs\") pod \"network-metrics-daemon-s98xl\" (UID: \"d45d9164-2ba4-400d-8b27-273d7f4f351d\") " pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:08:28.880812 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.880728 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txdkn\" (UniqueName: \"kubernetes.io/projected/d45d9164-2ba4-400d-8b27-273d7f4f351d-kube-api-access-txdkn\") pod \"network-metrics-daemon-s98xl\" (UID: \"d45d9164-2ba4-400d-8b27-273d7f4f351d\") " pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:08:28.880859 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.880839 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-modprobe-d\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.880903 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.880882 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-sysconfig\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.881122 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.881101 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5sx2\" (UniqueName: \"kubernetes.io/projected/d3acf985-6085-4061-8ca8-e6d7036cb6fd-kube-api-access-x5sx2\") pod \"kas-connection-checker-685895c7c9-5zc9z\" (UID: \"d3acf985-6085-4061-8ca8-e6d7036cb6fd\") " pod="kube-system/kas-connection-checker-685895c7c9-5zc9z" Nov 02 21:08:28.881206 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.881164 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-cnibin\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.881206 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.881196 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be519c9c-2677-4c4a-99c5-c537317bda3e-hosts-file\") pod \"node-resolver-gxdgr\" (UID: \"be519c9c-2677-4c4a-99c5-c537317bda3e\") " pod="openshift-dns/node-resolver-gxdgr" Nov 02 21:08:28.881299 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.881262 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-var-lib-openvswitch\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.881384 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.881355 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-sysctl-d\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.881432 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.881406 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-cnibin\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:08:28.881483 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.881445 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:08:28.881527 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.881492 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv27q\" (UniqueName: \"kubernetes.io/projected/c4a07e29-b928-4427-b4c2-5315984f03b3-kube-api-access-qv27q\") pod \"kas-connection-checker-685895c7c9-bb7st\" (UID: \"c4a07e29-b928-4427-b4c2-5315984f03b3\") " pod="kube-system/kas-connection-checker-685895c7c9-bb7st" Nov 02 21:08:28.881750 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.881731 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-registration-dir\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:08:28.881810 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.881772 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-multus-cni-dir\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.881856 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.881802 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-var-lib-cni-multus\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.881856 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.881842 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-multus-conf-dir\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.881943 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.881871 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rchb2\" (UniqueName: \"kubernetes.io/projected/d992040e-0d1e-4227-befb-cfef1acd8aad-kube-api-access-rchb2\") pod \"iptables-alerter-zpkht\" (UID: \"d992040e-0d1e-4227-befb-cfef1acd8aad\") " pod="openshift-network-operator/iptables-alerter-zpkht" Nov 02 21:08:28.881943 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.881903 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-certificates\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:28.881943 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.881933 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vmqkq\" (UID: \"6b5655ce-43ca-48e4-9e4b-872283ccb85c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" Nov 02 21:08:28.882070 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.881967 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtj26\" (UniqueName: \"kubernetes.io/projected/8ca0bae8-7539-4b19-912c-4379064ecfa2-kube-api-access-gtj26\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.882070 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.881997 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-var-lib-kubelet\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.882070 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.882020 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-device-dir\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:08:28.882070 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.882049 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-slash\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.882317 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.882078 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-systemd\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.882317 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.882108 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a2820a3-6d1e-42d0-8a89-0e47f870abc4-trusted-ca\") pod \"console-operator-7cd56b8dbc-qd9n6\" (UID: \"9a2820a3-6d1e-42d0-8a89-0e47f870abc4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:28.882317 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.882192 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d992040e-0d1e-4227-befb-cfef1acd8aad-iptables-alerter-script\") pod \"iptables-alerter-zpkht\" (UID: \"d992040e-0d1e-4227-befb-cfef1acd8aad\") " pod="openshift-network-operator/iptables-alerter-zpkht" Nov 02 21:08:28.882317 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.882220 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.882603 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.882584 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-run\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.882662 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.882635 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-sys\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.882707 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.882678 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/24c2ea1f-b61b-4e62-914a-2c52508bc04d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-qpklq\" (UID: \"24c2ea1f-b61b-4e62-914a-2c52508bc04d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qpklq" Nov 02 21:08:28.882750 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.882729 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6wd7\" (UniqueName: \"kubernetes.io/projected/bff8d097-26a3-4621-ac01-c3b00ed23e06-kube-api-access-w6wd7\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.882861 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.882845 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:28.882926 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.882885 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxcnw\" (UniqueName: \"kubernetes.io/projected/44a209b7-2144-45a0-bea0-2f8adf6658cb-kube-api-access-hxcnw\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:28.882926 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.882918 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cf9df1a4-3684-464c-97f3-21a3151a7183-image-registry-private-configuration\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:28.883029 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.882945 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-kubelet-config\") pod \"global-pull-secret-syncer-bllxh\" (UID: \"87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1\") " pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:08:28.883029 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.882974 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-kubelet\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.883029 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.883005 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-tuned\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.883184 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.883035 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-multus-socket-dir-parent\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.883184 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.883172 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-tuned\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.883277 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.883203 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgxs2\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-kube-api-access-kgxs2\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:28.883277 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.883258 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-run-netns\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.883366 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.883310 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bff8d097-26a3-4621-ac01-c3b00ed23e06-ovnkube-script-lib\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.883366 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.883350 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-default-certificate\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:28.883449 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.883376 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ab0662d-ef78-415d-98f6-8b271e23e16f-agent-certs\") pod \"konnectivity-agent-mz5qm\" (UID: \"3ab0662d-ef78-415d-98f6-8b271e23e16f\") " pod="kube-system/konnectivity-agent-mz5qm" Nov 02 21:08:28.883449 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.883416 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v2wr\" (UniqueName: \"kubernetes.io/projected/8cf65e09-2079-40cc-b597-1b67ba482a04-kube-api-access-5v2wr\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:28.883553 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.883449 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f264bf41-80c4-4dda-a67e-6651b464655c-serving-cert\") pod \"service-ca-operator-544d749c8c-pfjlk\" (UID: \"f264bf41-80c4-4dda-a67e-6651b464655c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pfjlk" Nov 02 21:08:28.883553 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.883479 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cf65e09-2079-40cc-b597-1b67ba482a04-tmp\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:28.883553 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.883512 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-original-pull-secret\") pod \"global-pull-secret-syncer-bllxh\" (UID: \"87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1\") " pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:08:28.883553 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.883543 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bff8d097-26a3-4621-ac01-c3b00ed23e06-env-overrides\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.883672 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.883579 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ca28f2b-c77d-4aa1-8501-ec380253ab88-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf\" (UID: \"5ca28f2b-c77d-4aa1-8501-ec380253ab88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf" Nov 02 21:08:28.883672 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.883612 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-lib-modules\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.883672 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.883641 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-etc-selinux\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:08:28.883754 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.883670 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08b38b87-8fd7-475e-94db-8bdd1339dc4b-trusted-ca\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:28.883754 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.883698 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08b38b87-8fd7-475e-94db-8bdd1339dc4b-installation-pull-secrets\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:28.883754 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.883733 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-systemd-units\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.883839 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.883825 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cf65e09-2079-40cc-b597-1b67ba482a04-tmp\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:28.883886 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.883852 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-cni-bin\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.883886 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.883872 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-crio-socket\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:28.883974 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.883893 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8cf65e09-2079-40cc-b597-1b67ba482a04-snapshots\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:28.883974 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.883934 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d49ad8b8-08aa-4da0-9139-b2be1efeb6cd-serviceca\") pod \"node-ca-77xn9\" (UID: \"d49ad8b8-08aa-4da0-9139-b2be1efeb6cd\") " pod="openshift-image-registry/node-ca-77xn9" Nov 02 21:08:28.884107 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.884088 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7kkw\" (UniqueName: \"kubernetes.io/projected/5ca28f2b-c77d-4aa1-8501-ec380253ab88-kube-api-access-t7kkw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf\" (UID: \"5ca28f2b-c77d-4aa1-8501-ec380253ab88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf" Nov 02 21:08:28.884171 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.884120 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-run-netns\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.884226 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.884200 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-etc-openvswitch\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.884292 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.884261 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad2a39a-7a74-44a2-a97f-88e84e65616c-config-volume\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:28.884385 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.884365 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8cf65e09-2079-40cc-b597-1b67ba482a04-snapshots\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:28.884436 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.884370 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxm2l\" (UniqueName: \"kubernetes.io/projected/7de0fc42-d787-49e7-80a2-5122a67bfba0-kube-api-access-jxm2l\") pod \"ingress-canary-mlgzw\" (UID: \"7de0fc42-d787-49e7-80a2-5122a67bfba0\") " pod="openshift-ingress-canary/ingress-canary-mlgzw" Nov 02 21:08:28.884771 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.884755 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-data-volume\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:28.884821 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.884788 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:28.884821 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.884813 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ad2a39a-7a74-44a2-a97f-88e84e65616c-tmp-dir\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:28.884910 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.884823 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-data-volume\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:28.884910 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.884828 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08b38b87-8fd7-475e-94db-8bdd1339dc4b-installation-pull-secrets\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:28.884910 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.884831 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-host\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.884998 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.884922 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ad2a39a-7a74-44a2-a97f-88e84e65616c-tmp-dir\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:28.884998 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.884948 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9a2820a3-6d1e-42d0-8a89-0e47f870abc4-config\") pod \"console-operator-7cd56b8dbc-qd9n6\" (UID: \"9a2820a3-6d1e-42d0-8a89-0e47f870abc4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:28.884998 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.884974 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cf65e09-2079-40cc-b597-1b67ba482a04-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:28.885098 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.884996 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-system-cni-dir\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.885098 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.885021 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-run-systemd\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.885098 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.885063 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:28.885209 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.885101 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a899c9c5-79b4-472d-942c-5e0c4c53e954-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-z4kx6\" (UID: \"a899c9c5-79b4-472d-942c-5e0c4c53e954\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6" Nov 02 21:08:28.885209 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.885148 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d992040e-0d1e-4227-befb-cfef1acd8aad-host-slash\") pod \"iptables-alerter-zpkht\" (UID: \"d992040e-0d1e-4227-befb-cfef1acd8aad\") " pod="openshift-network-operator/iptables-alerter-zpkht" Nov 02 21:08:28.885209 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.885201 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-run-openvswitch\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.885299 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.885222 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:08:28.885299 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.885237 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-run-k8s-cni-cncf-io\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.885299 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.885254 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:28.885299 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.885276 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08b38b87-8fd7-475e-94db-8bdd1339dc4b-image-registry-private-configuration\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:28.885409 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.885314 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-socket-dir\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:08:28.885409 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.885370 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-etc-kubernetes\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.885409 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.885405 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fshk5\" (UniqueName: \"kubernetes.io/projected/be519c9c-2677-4c4a-99c5-c537317bda3e-kube-api-access-fshk5\") pod \"node-resolver-gxdgr\" (UID: \"be519c9c-2677-4c4a-99c5-c537317bda3e\") " pod="openshift-dns/node-resolver-gxdgr" Nov 02 21:08:28.885492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.885438 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-cni-netd\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.885492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.885456 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmqww\" (UniqueName: \"kubernetes.io/projected/4ad2a39a-7a74-44a2-a97f-88e84e65616c-kube-api-access-fmqww\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:28.885492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.885474 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-sys-fs\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:08:28.885681 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.885495 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bff8d097-26a3-4621-ac01-c3b00ed23e06-ovn-node-metrics-cert\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.885681 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.885553 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx8dl\" (UniqueName: \"kubernetes.io/projected/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-kube-api-access-jx8dl\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:28.885681 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.885667 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-sysctl-conf\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.885788 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.885710 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7204d98-9f92-46e6-9e6d-482371e0a3ee-tmp\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.885788 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.885745 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnvnx\" (UniqueName: \"kubernetes.io/projected/9a2820a3-6d1e-42d0-8a89-0e47f870abc4-kube-api-access-gnvnx\") pod \"console-operator-7cd56b8dbc-qd9n6\" (UID: \"9a2820a3-6d1e-42d0-8a89-0e47f870abc4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:28.885788 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.885761 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7204d98-9f92-46e6-9e6d-482371e0a3ee-tmp\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.885788 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.885778 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vmqkq\" (UID: \"6b5655ce-43ca-48e4-9e4b-872283ccb85c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" Nov 02 21:08:28.885927 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.885797 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-var-lib-cni-bin\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.885927 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.885815 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-run-multus-certs\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.905302 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.905277 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 21:08:28.905551 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.905535 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08b38b87-8fd7-475e-94db-8bdd1339dc4b-image-registry-private-configuration\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:28.905719 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.905702 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ftl5q\"" Nov 02 21:08:28.905780 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.905715 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 02 21:08:28.905780 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.905720 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 21:08:28.905847 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.905796 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-97nlp\"" Nov 02 21:08:28.910625 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.910611 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9a2820a3-6d1e-42d0-8a89-0e47f870abc4-serving-cert\") pod \"console-operator-7cd56b8dbc-qd9n6\" (UID: \"9a2820a3-6d1e-42d0-8a89-0e47f870abc4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:28.911117 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.911093 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 02 21:08:28.911236 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.911146 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 02 21:08:28.911297 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.911281 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 02 21:08:28.911349 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.911287 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 02 21:08:28.911422 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.911379 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-r64zg\"" Nov 02 21:08:28.911422 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.911386 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 02 21:08:28.911527 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.911451 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9vsgv\"" Nov 02 21:08:28.911617 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.911595 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 21:08:28.911666 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.911607 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-sntml\"" Nov 02 21:08:28.911666 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.911644 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 21:08:28.911739 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.911678 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-t7psx\"" Nov 02 21:08:28.911739 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.911723 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-894d2\"" Nov 02 21:08:28.911824 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.911759 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 21:08:28.911824 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.911799 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9p4p6\"" Nov 02 21:08:28.911824 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.911821 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vvdb6\"" Nov 02 21:08:28.912355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.912337 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 21:08:28.912494 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.912399 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 21:08:28.912616 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.912601 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-58fxm\"" Nov 02 21:08:28.912616 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.912607 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rbqzp\"" Nov 02 21:08:28.912717 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.912639 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 02 21:08:28.912717 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.912648 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 02 21:08:28.912717 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.912669 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jb7zz\"" Nov 02 21:08:28.912717 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.912708 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f8dkx\"" Nov 02 21:08:28.912979 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.912966 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-ml487\"" Nov 02 21:08:28.913038 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.913020 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-m78tx\"" Nov 02 21:08:28.913087 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.913025 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wnfwz\"" Nov 02 21:08:28.913087 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.913074 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zzvdw\"" Nov 02 21:08:28.913218 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.913204 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 21:08:28.913271 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.913206 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fx9vz\"" Nov 02 21:08:28.913271 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.913203 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-z9mmm\"" Nov 02 21:08:28.913351 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.913288 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-d9psf\"" Nov 02 21:08:28.913351 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.913300 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9rn4m\"" Nov 02 21:08:28.913351 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.913301 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-v96df\"" Nov 02 21:08:28.913351 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.913347 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 21:08:28.913520 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.913400 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 21:08:28.913520 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.913463 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2mhzd\"" Nov 02 21:08:28.913520 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.913466 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cf9df1a4-3684-464c-97f3-21a3151a7183-image-registry-private-configuration\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:28.913520 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.913496 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9hv44\"" Nov 02 21:08:28.914778 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.914762 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f264bf41-80c4-4dda-a67e-6651b464655c-serving-cert\") pod \"service-ca-operator-544d749c8c-pfjlk\" (UID: \"f264bf41-80c4-4dda-a67e-6651b464655c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pfjlk" Nov 02 21:08:28.914858 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.914778 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-default-certificate\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:28.914916 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.914856 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-original-pull-secret\") pod \"global-pull-secret-syncer-bllxh\" (UID: \"87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1\") " pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:08:28.914964 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.914943 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ab0662d-ef78-415d-98f6-8b271e23e16f-agent-certs\") pod \"konnectivity-agent-mz5qm\" (UID: \"3ab0662d-ef78-415d-98f6-8b271e23e16f\") " pod="kube-system/konnectivity-agent-mz5qm" Nov 02 21:08:28.915972 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.915953 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bff8d097-26a3-4621-ac01-c3b00ed23e06-ovn-node-metrics-cert\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.917338 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.917320 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 02 21:08:28.917454 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.917438 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 02 21:08:28.917769 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.917750 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 21:08:28.917769 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.917763 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 21:08:28.917892 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.917788 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 02 21:08:28.917892 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.917874 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:08:28.917982 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.917918 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 21:08:28.917982 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.917974 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 21:08:28.918048 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.917972 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 02 21:08:28.918094 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.918075 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 21:08:28.918161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.918103 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 21:08:28.918213 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.918162 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:08:28.918213 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.918202 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 21:08:28.918311 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.918210 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 21:08:28.918311 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.918179 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 21:08:28.918311 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.918178 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 21:08:28.918311 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.918232 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 21:08:28.918311 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.918236 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 02 21:08:28.918311 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.918247 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 02 21:08:28.918311 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.918288 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 21:08:28.918617 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.918325 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 21:08:28.918617 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.918340 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 21:08:28.918617 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.918341 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 21:08:28.918617 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.918407 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 21:08:28.918617 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.918413 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:08:28.918617 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.918432 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 21:08:28.918617 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.918513 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 21:08:28.918617 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.918556 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 21:08:28.918824 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.918631 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 02 21:08:28.918824 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.918801 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-stats-auth\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:28.918982 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.918968 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7de0fc42-d787-49e7-80a2-5122a67bfba0-cert\") pod \"ingress-canary-mlgzw\" (UID: \"7de0fc42-d787-49e7-80a2-5122a67bfba0\") " pod="openshift-ingress-canary/ingress-canary-mlgzw" Nov 02 21:08:28.919121 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.919107 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 21:08:28.920699 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.920686 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-certificates\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:28.920738 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.920717 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ca28f2b-c77d-4aa1-8501-ec380253ab88-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf\" (UID: \"5ca28f2b-c77d-4aa1-8501-ec380253ab88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf" Nov 02 21:08:28.920771 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.920747 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f264bf41-80c4-4dda-a67e-6651b464655c-config\") pod \"service-ca-operator-544d749c8c-pfjlk\" (UID: \"f264bf41-80c4-4dda-a67e-6651b464655c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pfjlk" Nov 02 21:08:28.921923 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.921908 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bff8d097-26a3-4621-ac01-c3b00ed23e06-ovnkube-config\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.923037 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.923025 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-certificates\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:28.923155 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.923122 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d992040e-0d1e-4227-befb-cfef1acd8aad-iptables-alerter-script\") pod \"iptables-alerter-zpkht\" (UID: \"d992040e-0d1e-4227-befb-cfef1acd8aad\") " pod="openshift-network-operator/iptables-alerter-zpkht" Nov 02 21:08:28.925270 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.925256 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad2a39a-7a74-44a2-a97f-88e84e65616c-config-volume\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:28.925322 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.925307 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d49ad8b8-08aa-4da0-9139-b2be1efeb6cd-serviceca\") pod \"node-ca-77xn9\" (UID: \"d49ad8b8-08aa-4da0-9139-b2be1efeb6cd\") " pod="openshift-image-registry/node-ca-77xn9" Nov 02 21:08:28.926359 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.926345 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vmqkq\" (UID: \"6b5655ce-43ca-48e4-9e4b-872283ccb85c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" Nov 02 21:08:28.928506 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.928491 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ab0662d-ef78-415d-98f6-8b271e23e16f-konnectivity-ca\") pod \"konnectivity-agent-mz5qm\" (UID: \"3ab0662d-ef78-415d-98f6-8b271e23e16f\") " pod="kube-system/konnectivity-agent-mz5qm" Nov 02 21:08:28.928571 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.928557 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8ca0bae8-7539-4b19-912c-4379064ecfa2-multus-daemon-config\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.952367 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.952336 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 21:08:28.952503 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.952340 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 21:08:28.952503 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.952441 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 02 21:08:28.952503 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.952468 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 02 21:08:28.952649 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.952619 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 02 21:08:28.952649 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.952630 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 21:08:28.952740 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.952696 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 21:08:28.952740 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.952701 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 21:08:28.952740 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.952706 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 21:08:28.952740 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.952704 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 21:08:28.952740 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.952726 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 21:08:28.952740 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.952712 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 21:08:28.954469 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.954453 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ca28f2b-c77d-4aa1-8501-ec380253ab88-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf\" (UID: \"5ca28f2b-c77d-4aa1-8501-ec380253ab88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf" Nov 02 21:08:28.954587 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.954576 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bff8d097-26a3-4621-ac01-c3b00ed23e06-env-overrides\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.955553 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.955537 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9a2820a3-6d1e-42d0-8a89-0e47f870abc4-config\") pod \"console-operator-7cd56b8dbc-qd9n6\" (UID: \"9a2820a3-6d1e-42d0-8a89-0e47f870abc4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:28.957828 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.957812 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 21:08:28.957876 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.957819 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 02 21:08:28.957876 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.957832 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 21:08:28.957876 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.957847 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 21:08:28.957876 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.957867 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 21:08:28.958103 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.958088 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 21:08:28.958246 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.958156 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:08:28.958246 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.958179 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:08:28.958246 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.958182 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 02 21:08:28.958338 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.958263 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:08:28.958338 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.958288 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 02 21:08:28.958396 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.958340 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 21:08:28.958396 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.958345 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:08:28.958396 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.958373 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 21:08:28.959054 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.959036 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ca0bae8-7539-4b19-912c-4379064ecfa2-cni-binary-copy\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.959091 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.959036 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:08:28.959507 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.959485 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cf65e09-2079-40cc-b597-1b67ba482a04-service-ca-bundle\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:28.960071 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.960034 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:28.960176 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.960145 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 21:08:28.961425 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.961408 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:08:28.962109 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.962092 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:08:28.963285 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.963269 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/24c2ea1f-b61b-4e62-914a-2c52508bc04d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-qpklq\" (UID: \"24c2ea1f-b61b-4e62-914a-2c52508bc04d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qpklq" Nov 02 21:08:28.963764 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.963749 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bff8d097-26a3-4621-ac01-c3b00ed23e06-ovnkube-script-lib\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.964225 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.964211 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 02 21:08:28.966010 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.965387 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfhs6\" (UniqueName: \"kubernetes.io/projected/345370c7-0ca8-4afe-94db-c9f5394d4d72-kube-api-access-rfhs6\") pod \"kas-connection-checker-685895c7c9-nv2sw\" (UID: \"345370c7-0ca8-4afe-94db-c9f5394d4d72\") " pod="kube-system/kas-connection-checker-685895c7c9-nv2sw" Nov 02 21:08:28.966010 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.965463 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq5t9\" (UniqueName: \"kubernetes.io/projected/42b83ba5-5e98-459f-ad4f-6a94c90212be-kube-api-access-lq5t9\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:08:28.966010 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.965478 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx42g\" (UniqueName: \"kubernetes.io/projected/a7204d98-9f92-46e6-9e6d-482371e0a3ee-kube-api-access-sx42g\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.966010 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.965500 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kmdm\" (UniqueName: \"kubernetes.io/projected/39830cb0-c259-446d-b2d3-fd8956b968fb-kube-api-access-9kmdm\") pod \"migrator-66df5d496b-wf84d\" (UID: \"39830cb0-c259-446d-b2d3-fd8956b968fb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wf84d" Nov 02 21:08:28.966010 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.965530 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxbh5\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-kube-api-access-bxbh5\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:28.966010 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.965578 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgqft\" (UniqueName: \"kubernetes.io/projected/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-kube-api-access-wgqft\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:08:28.966010 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.965600 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gttn\" (UniqueName: \"kubernetes.io/projected/a899c9c5-79b4-472d-942c-5e0c4c53e954-kube-api-access-9gttn\") pod \"cluster-monitoring-operator-884576c57-z4kx6\" (UID: \"a899c9c5-79b4-472d-942c-5e0c4c53e954\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6" Nov 02 21:08:28.966010 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.965665 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w7df\" (UniqueName: \"kubernetes.io/projected/fecc158a-a288-4064-b3e9-6fe91e703f5b-kube-api-access-7w7df\") pod \"network-check-source-6f78d59f99-8jclh\" (UID: \"fecc158a-a288-4064-b3e9-6fe91e703f5b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-8jclh" Nov 02 21:08:28.966010 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.965815 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a899c9c5-79b4-472d-942c-5e0c4c53e954-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-z4kx6\" (UID: \"a899c9c5-79b4-472d-942c-5e0c4c53e954\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6" Nov 02 21:08:28.966010 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.965920 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-bound-sa-token\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:28.966010 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.965965 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqv42\" (UniqueName: \"kubernetes.io/projected/d49ad8b8-08aa-4da0-9139-b2be1efeb6cd-kube-api-access-gqv42\") pod \"node-ca-77xn9\" (UID: \"d49ad8b8-08aa-4da0-9139-b2be1efeb6cd\") " pod="openshift-image-registry/node-ca-77xn9" Nov 02 21:08:28.966010 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.966007 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q672k\" (UniqueName: \"kubernetes.io/projected/6b5655ce-43ca-48e4-9e4b-872283ccb85c-kube-api-access-q672k\") pod \"cluster-samples-operator-6ddb6c8b94-vmqkq\" (UID: \"6b5655ce-43ca-48e4-9e4b-872283ccb85c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" Nov 02 21:08:28.966429 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.966030 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw9lp\" (UniqueName: \"kubernetes.io/projected/f264bf41-80c4-4dda-a67e-6651b464655c-kube-api-access-gw9lp\") pod \"service-ca-operator-544d749c8c-pfjlk\" (UID: \"f264bf41-80c4-4dda-a67e-6651b464655c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pfjlk" Nov 02 21:08:28.966429 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.966066 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx9qw\" (UniqueName: \"kubernetes.io/projected/934bd6cf-3dcb-40fd-9b11-97886b3f098b-kube-api-access-vx9qw\") pod \"volume-data-source-validator-58c9f5f46d-qpf24\" (UID: \"934bd6cf-3dcb-40fd-9b11-97886b3f098b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qpf24" Nov 02 21:08:28.986869 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.986843 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-lib-modules\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.986955 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.986872 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-etc-selinux\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:08:28.986955 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.986889 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-systemd-units\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.986955 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.986915 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-cni-bin\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.986955 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.986939 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-crio-socket\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:28.987152 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.986966 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-run-netns\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.987152 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.986977 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-systemd-units\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.987152 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.986983 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-etc-selinux\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:08:28.987152 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987006 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-run-netns\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.987152 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987003 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-etc-openvswitch\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.987152 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987039 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:28.987152 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.986986 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-cni-bin\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.987152 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987056 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-lib-modules\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.987152 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987047 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-crio-socket\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:28.987152 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987059 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-host\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.987152 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987049 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-etc-openvswitch\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.987152 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987102 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-host\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.987152 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987108 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-system-cni-dir\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987178 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-run-systemd\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:28.987189 4935 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987209 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-system-cni-dir\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987236 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987252 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d992040e-0d1e-4227-befb-cfef1acd8aad-host-slash\") pod \"iptables-alerter-zpkht\" (UID: \"d992040e-0d1e-4227-befb-cfef1acd8aad\") " pod="openshift-network-operator/iptables-alerter-zpkht" Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987279 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-run-systemd\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987303 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-run-openvswitch\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987297 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d992040e-0d1e-4227-befb-cfef1acd8aad-host-slash\") pod \"iptables-alerter-zpkht\" (UID: \"d992040e-0d1e-4227-befb-cfef1acd8aad\") " pod="openshift-network-operator/iptables-alerter-zpkht" Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:28.987315 4935 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:28.987385 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls podName:4ad2a39a-7a74-44a2-a97f-88e84e65616c nodeName:}" failed. No retries permitted until 2026-11-02 21:08:29.487359788 +0000 UTC m=+1.926934508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls") pod "dns-default-vn62q" (UID: "4ad2a39a-7a74-44a2-a97f-88e84e65616c") : secret "dns-default-metrics-tls" not found Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987389 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-run-openvswitch\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:28.987407 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs podName:44a209b7-2144-45a0-bea0-2f8adf6658cb nodeName:}" failed. No retries permitted until 2026-11-02 21:08:29.48739824 +0000 UTC m=+1.926972943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs") pod "router-default-798b7b6b45-hlcl7" (UID: "44a209b7-2144-45a0-bea0-2f8adf6658cb") : secret "router-metrics-certs-default" not found Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987429 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987460 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-run-k8s-cni-cncf-io\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987469 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987504 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987525 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-run-k8s-cni-cncf-io\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987534 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-socket-dir\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987558 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-etc-kubernetes\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987582 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-cni-netd\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987598 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-socket-dir\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987607 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-sys-fs\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987640 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-cni-netd\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987642 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-sysctl-conf\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987658 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-sys-fs\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987674 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-var-lib-cni-bin\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:28.987678 4935 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:28.987691 4935 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c987dc54-2p2h6: secret "image-registry-tls" not found Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987702 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-var-lib-cni-bin\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987709 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-run-multus-certs\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987690 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-run-multus-certs\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.987723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987742 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-etc-kubernetes\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:28.987748 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls podName:cf9df1a4-3684-464c-97f3-21a3151a7183 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:29.487731704 +0000 UTC m=+1.927306424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls") pod "image-registry-c987dc54-2p2h6" (UID: "cf9df1a4-3684-464c-97f3-21a3151a7183") : secret "image-registry-tls" not found Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987759 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-sysctl-conf\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987773 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987796 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-dbus\") pod \"global-pull-secret-syncer-bllxh\" (UID: \"87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1\") " pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987816 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-z4kx6\" (UID: \"a899c9c5-79b4-472d-942c-5e0c4c53e954\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:28.987825 4935 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:28.987837 4935 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-588654849f-w89ht: secret "image-registry-tls" not found Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987863 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d49ad8b8-08aa-4da0-9139-b2be1efeb6cd-host\") pod \"node-ca-77xn9\" (UID: \"d49ad8b8-08aa-4da0-9139-b2be1efeb6cd\") " pod="openshift-image-registry/node-ca-77xn9" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:28.987876 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls podName:08b38b87-8fd7-475e-94db-8bdd1339dc4b nodeName:}" failed. No retries permitted until 2026-11-02 21:08:29.487866397 +0000 UTC m=+1.927441110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls") pod "image-registry-588654849f-w89ht" (UID: "08b38b87-8fd7-475e-94db-8bdd1339dc4b") : secret "image-registry-tls" not found Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987833 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d49ad8b8-08aa-4da0-9139-b2be1efeb6cd-host\") pod \"node-ca-77xn9\" (UID: \"d49ad8b8-08aa-4da0-9139-b2be1efeb6cd\") " pod="openshift-image-registry/node-ca-77xn9" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987883 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-dbus\") pod \"global-pull-secret-syncer-bllxh\" (UID: \"87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1\") " pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987912 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-log-socket\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987927 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-log-socket\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987933 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-run-ovn\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:28.987961 4935 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987958 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-run-ovn\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:28.987996 4935 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:28.988008 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls podName:a899c9c5-79b4-472d-942c-5e0c4c53e954 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:29.487994637 +0000 UTC m=+1.927569339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-z4kx6" (UID: "a899c9c5-79b4-472d-942c-5e0c4c53e954") : secret "cluster-monitoring-operator-tls" not found Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.987960 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qpklq\" (UID: \"24c2ea1f-b61b-4e62-914a-2c52508bc04d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qpklq" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:28.988034 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert podName:24c2ea1f-b61b-4e62-914a-2c52508bc04d nodeName:}" failed. No retries permitted until 2026-11-02 21:08:29.488024652 +0000 UTC m=+1.927599356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qpklq" (UID: "24c2ea1f-b61b-4e62-914a-2c52508bc04d") : secret "networking-console-plugin-cert" not found Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988050 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-system-cni-dir\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988065 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-hostroot\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988084 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-var-lib-kubelet\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988101 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988120 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988145 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-hostroot\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988148 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-kubernetes\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988160 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-var-lib-kubelet\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988188 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988194 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988203 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-system-cni-dir\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:28.988212 4935 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988211 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-run-ovn-kubernetes\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:28.988244 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls podName:67c3caa0-0b53-42c9-98b2-96addd3dd7fc nodeName:}" failed. No retries permitted until 2026-11-02 21:08:29.488236691 +0000 UTC m=+1.927811420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2nshl" (UID: "67c3caa0-0b53-42c9-98b2-96addd3dd7fc") : secret "insights-runtime-extractor-tls" not found Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988269 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-node-log\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988207 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-kubernetes\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988301 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs\") pod \"network-metrics-daemon-s98xl\" (UID: \"d45d9164-2ba4-400d-8b27-273d7f4f351d\") " pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988315 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-run-ovn-kubernetes\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988321 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-modprobe-d\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988339 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-sysconfig\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988355 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-cnibin\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:28.988361 4935 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988369 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be519c9c-2677-4c4a-99c5-c537317bda3e-hosts-file\") pod \"node-resolver-gxdgr\" (UID: \"be519c9c-2677-4c4a-99c5-c537317bda3e\") " pod="openshift-dns/node-resolver-gxdgr" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988395 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-var-lib-openvswitch\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:28.988400 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs podName:d45d9164-2ba4-400d-8b27-273d7f4f351d nodeName:}" failed. No retries permitted until 2026-11-02 21:08:29.488391878 +0000 UTC m=+1.927966600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs") pod "network-metrics-daemon-s98xl" (UID: "d45d9164-2ba4-400d-8b27-273d7f4f351d") : secret "metrics-daemon-secret" not found Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988411 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-node-log\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988433 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-sysctl-d\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988438 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-sysconfig\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988442 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be519c9c-2677-4c4a-99c5-c537317bda3e-hosts-file\") pod \"node-resolver-gxdgr\" (UID: \"be519c9c-2677-4c4a-99c5-c537317bda3e\") " pod="openshift-dns/node-resolver-gxdgr" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988458 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-cnibin\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988463 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-var-lib-openvswitch\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988465 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-cnibin\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988481 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-registration-dir\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988506 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-multus-cni-dir\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988518 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7dab1353-88e8-4250-b7ab-a4b2a69e37fa-cnibin\") pod \"multus-additional-cni-plugins-94zch\" (UID: \"7dab1353-88e8-4250-b7ab-a4b2a69e37fa\") " pod="openshift-multus/multus-additional-cni-plugins-94zch" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988546 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-var-lib-cni-multus\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988519 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-registration-dir\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988522 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-modprobe-d\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988559 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-multus-cni-dir\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988566 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-sysctl-d\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988579 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-multus-conf-dir\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988613 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vmqkq\" (UID: \"6b5655ce-43ca-48e4-9e4b-872283ccb85c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988613 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-multus-conf-dir\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988584 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-host-var-lib-cni-multus\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988644 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-var-lib-kubelet\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988668 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-device-dir\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:28.988685 4935 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988696 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-slash\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:28.988711 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls podName:6b5655ce-43ca-48e4-9e4b-872283ccb85c nodeName:}" failed. No retries permitted until 2026-11-02 21:08:29.488703177 +0000 UTC m=+1.928277877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vmqkq" (UID: "6b5655ce-43ca-48e4-9e4b-872283ccb85c") : secret "samples-operator-tls" not found Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988725 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-var-lib-kubelet\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988732 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-slash\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988731 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-systemd\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988758 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988773 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-etc-systemd\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988764 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/42b83ba5-5e98-459f-ad4f-6a94c90212be-device-dir\") pod \"aws-ebs-csi-driver-node-hnm4n\" (UID: \"42b83ba5-5e98-459f-ad4f-6a94c90212be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hnm4n" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988774 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988814 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988826 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-run\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988850 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-sys\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988861 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-run\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988878 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988908 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-kubelet-config\") pod \"global-pull-secret-syncer-bllxh\" (UID: \"87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1\") " pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988915 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7204d98-9f92-46e6-9e6d-482371e0a3ee-sys\") pod \"tuned-z5hfn\" (UID: \"a7204d98-9f92-46e6-9e6d-482371e0a3ee\") " pod="openshift-cluster-node-tuning-operator/tuned-z5hfn" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988936 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-kubelet\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.989022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.988962 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-multus-socket-dir-parent\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.991380 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:28.988972 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle podName:44a209b7-2144-45a0-bea0-2f8adf6658cb nodeName:}" failed. No retries permitted until 2026-11-02 21:08:29.488962112 +0000 UTC m=+1.928536841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle") pod "router-default-798b7b6b45-hlcl7" (UID: "44a209b7-2144-45a0-bea0-2f8adf6658cb") : configmap references non-existent config key: service-ca.crt Nov 02 21:08:28.991380 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.989005 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ca0bae8-7539-4b19-912c-4379064ecfa2-multus-socket-dir-parent\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:28.991380 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.989014 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-run-netns\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.991380 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.989026 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1-kubelet-config\") pod \"global-pull-secret-syncer-bllxh\" (UID: \"87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1\") " pod="kube-system/global-pull-secret-syncer-bllxh" Nov 02 21:08:28.991380 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.989045 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-run-netns\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:28.991380 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:28.989010 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bff8d097-26a3-4621-ac01-c3b00ed23e06-host-kubelet\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:29.004626 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.004606 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 21:08:29.004712 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.004693 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 21:08:29.004823 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.004802 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 02 21:08:29.005236 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.005214 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-bound-sa-token\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:29.005304 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.005262 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txdkn\" (UniqueName: \"kubernetes.io/projected/d45d9164-2ba4-400d-8b27-273d7f4f351d-kube-api-access-txdkn\") pod \"network-metrics-daemon-s98xl\" (UID: \"d45d9164-2ba4-400d-8b27-273d7f4f351d\") " pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:08:29.005359 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.005316 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5sx2\" (UniqueName: \"kubernetes.io/projected/d3acf985-6085-4061-8ca8-e6d7036cb6fd-kube-api-access-x5sx2\") pod \"kas-connection-checker-685895c7c9-5zc9z\" (UID: \"d3acf985-6085-4061-8ca8-e6d7036cb6fd\") " pod="kube-system/kas-connection-checker-685895c7c9-5zc9z" Nov 02 21:08:29.005436 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.005411 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rchb2\" (UniqueName: \"kubernetes.io/projected/d992040e-0d1e-4227-befb-cfef1acd8aad-kube-api-access-rchb2\") pod \"iptables-alerter-zpkht\" (UID: \"d992040e-0d1e-4227-befb-cfef1acd8aad\") " pod="openshift-network-operator/iptables-alerter-zpkht" Nov 02 21:08:29.005630 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.005606 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7kkw\" (UniqueName: \"kubernetes.io/projected/5ca28f2b-c77d-4aa1-8501-ec380253ab88-kube-api-access-t7kkw\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf\" (UID: \"5ca28f2b-c77d-4aa1-8501-ec380253ab88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf" Nov 02 21:08:29.005630 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.005621 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxcnw\" (UniqueName: \"kubernetes.io/projected/44a209b7-2144-45a0-bea0-2f8adf6658cb-kube-api-access-hxcnw\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:29.005772 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.005754 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtj26\" (UniqueName: \"kubernetes.io/projected/8ca0bae8-7539-4b19-912c-4379064ecfa2-kube-api-access-gtj26\") pod \"multus-6blt8\" (UID: \"8ca0bae8-7539-4b19-912c-4379064ecfa2\") " pod="openshift-multus/multus-6blt8" Nov 02 21:08:29.005920 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.005896 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6wd7\" (UniqueName: \"kubernetes.io/projected/bff8d097-26a3-4621-ac01-c3b00ed23e06-kube-api-access-w6wd7\") pod \"ovnkube-node-pqnmg\" (UID: \"bff8d097-26a3-4621-ac01-c3b00ed23e06\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:29.005983 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.005968 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v2wr\" (UniqueName: \"kubernetes.io/projected/8cf65e09-2079-40cc-b597-1b67ba482a04-kube-api-access-5v2wr\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:29.006095 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.006075 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfbgk\" (UniqueName: \"kubernetes.io/projected/c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7-kube-api-access-lfbgk\") pod \"network-check-target-8c67p\" (UID: \"c96ef2e1-52b0-4f0d-8e28-0d1dfa8fdfb7\") " pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:08:29.006299 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.006277 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxm2l\" (UniqueName: \"kubernetes.io/projected/7de0fc42-d787-49e7-80a2-5122a67bfba0-kube-api-access-jxm2l\") pod \"ingress-canary-mlgzw\" (UID: \"7de0fc42-d787-49e7-80a2-5122a67bfba0\") " pod="openshift-ingress-canary/ingress-canary-mlgzw" Nov 02 21:08:29.006299 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.006287 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fshk5\" (UniqueName: \"kubernetes.io/projected/be519c9c-2677-4c4a-99c5-c537317bda3e-kube-api-access-fshk5\") pod \"node-resolver-gxdgr\" (UID: \"be519c9c-2677-4c4a-99c5-c537317bda3e\") " pod="openshift-dns/node-resolver-gxdgr" Nov 02 21:08:29.006486 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.006461 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnvnx\" (UniqueName: \"kubernetes.io/projected/9a2820a3-6d1e-42d0-8a89-0e47f870abc4-kube-api-access-gnvnx\") pod \"console-operator-7cd56b8dbc-qd9n6\" (UID: \"9a2820a3-6d1e-42d0-8a89-0e47f870abc4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:29.006544 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.006468 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgxs2\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-kube-api-access-kgxs2\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:29.006544 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.006469 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx8dl\" (UniqueName: \"kubernetes.io/projected/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-kube-api-access-jx8dl\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:29.006617 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.006578 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv27q\" (UniqueName: \"kubernetes.io/projected/c4a07e29-b928-4427-b4c2-5315984f03b3-kube-api-access-qv27q\") pod \"kas-connection-checker-685895c7c9-bb7st\" (UID: \"c4a07e29-b928-4427-b4c2-5315984f03b3\") " pod="kube-system/kas-connection-checker-685895c7c9-bb7st" Nov 02 21:08:29.008191 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.007292 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmqww\" (UniqueName: \"kubernetes.io/projected/4ad2a39a-7a74-44a2-a97f-88e84e65616c-kube-api-access-fmqww\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:29.008191 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.007391 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cf65e09-2079-40cc-b597-1b67ba482a04-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-2ftg6\" (UID: \"8cf65e09-2079-40cc-b597-1b67ba482a04\") " pod="openshift-insights/insights-operator-7c4986d449-2ftg6" Nov 02 21:08:29.010960 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.010934 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cf9df1a4-3684-464c-97f3-21a3151a7183-trusted-ca\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:29.013729 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.013706 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a2820a3-6d1e-42d0-8a89-0e47f870abc4-trusted-ca\") pod \"console-operator-7cd56b8dbc-qd9n6\" (UID: \"9a2820a3-6d1e-42d0-8a89-0e47f870abc4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:29.014892 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.014869 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08b38b87-8fd7-475e-94db-8bdd1339dc4b-trusted-ca\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:29.120398 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.120363 4935 scope.go:122] "RemoveContainer" containerID="25880bea858c09ebae11d9d77d74a3d9d1f8c364288f4a9ac3048abab9b30755" Nov 02 21:08:29.134979 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.134957 4935 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:08:29.169052 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.169029 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:29.493740 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.493652 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs\") pod \"network-metrics-daemon-s98xl\" (UID: \"d45d9164-2ba4-400d-8b27-273d7f4f351d\") " pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:08:29.493740 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.493694 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vmqkq\" (UID: \"6b5655ce-43ca-48e4-9e4b-872283ccb85c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" Nov 02 21:08:29.493740 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.493720 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:29.493991 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.493799 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:29.493991 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:29.493817 4935 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 21:08:29.493991 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:29.493829 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle podName:44a209b7-2144-45a0-bea0-2f8adf6658cb nodeName:}" failed. No retries permitted until 2026-11-02 21:08:30.493814976 +0000 UTC m=+2.933389676 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle") pod "router-default-798b7b6b45-hlcl7" (UID: "44a209b7-2144-45a0-bea0-2f8adf6658cb") : configmap references non-existent config key: service-ca.crt Nov 02 21:08:29.493991 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.493853 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:29.493991 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:29.493866 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls podName:6b5655ce-43ca-48e4-9e4b-872283ccb85c nodeName:}" failed. No retries permitted until 2026-11-02 21:08:30.493854935 +0000 UTC m=+2.933429672 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vmqkq" (UID: "6b5655ce-43ca-48e4-9e4b-872283ccb85c") : secret "samples-operator-tls" not found Nov 02 21:08:29.493991 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:29.493875 4935 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:08:29.493991 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.493899 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:29.493991 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:29.493910 4935 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 21:08:29.493991 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:29.493918 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls podName:4ad2a39a-7a74-44a2-a97f-88e84e65616c nodeName:}" failed. No retries permitted until 2026-11-02 21:08:30.493904449 +0000 UTC m=+2.933479163 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls") pod "dns-default-vn62q" (UID: "4ad2a39a-7a74-44a2-a97f-88e84e65616c") : secret "dns-default-metrics-tls" not found Nov 02 21:08:29.493991 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:29.493920 4935 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 21:08:29.493991 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:29.493944 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs podName:44a209b7-2144-45a0-bea0-2f8adf6658cb nodeName:}" failed. No retries permitted until 2026-11-02 21:08:30.493933235 +0000 UTC m=+2.933507939 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs") pod "router-default-798b7b6b45-hlcl7" (UID: "44a209b7-2144-45a0-bea0-2f8adf6658cb") : secret "router-metrics-certs-default" not found Nov 02 21:08:29.493991 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:29.493953 4935 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:08:29.493991 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:29.493964 4935 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c987dc54-2p2h6: secret "image-registry-tls" not found Nov 02 21:08:29.493991 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:29.493971 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs podName:d45d9164-2ba4-400d-8b27-273d7f4f351d nodeName:}" failed. No retries permitted until 2026-11-02 21:08:30.493960197 +0000 UTC m=+2.933534931 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs") pod "network-metrics-daemon-s98xl" (UID: "d45d9164-2ba4-400d-8b27-273d7f4f351d") : secret "metrics-daemon-secret" not found Nov 02 21:08:29.493991 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:29.493992 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls podName:cf9df1a4-3684-464c-97f3-21a3151a7183 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:30.493985378 +0000 UTC m=+2.933560114 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls") pod "image-registry-c987dc54-2p2h6" (UID: "cf9df1a4-3684-464c-97f3-21a3151a7183") : secret "image-registry-tls" not found Nov 02 21:08:29.494487 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.494005 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:29.494487 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.494027 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-z4kx6\" (UID: \"a899c9c5-79b4-472d-942c-5e0c4c53e954\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6" Nov 02 21:08:29.494487 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.494056 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qpklq\" (UID: \"24c2ea1f-b61b-4e62-914a-2c52508bc04d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qpklq" Nov 02 21:08:29.494487 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:29.494062 4935 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:08:29.494487 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:29.494070 4935 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-588654849f-w89ht: secret "image-registry-tls" not found Nov 02 21:08:29.494487 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.494081 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:29.494487 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:29.494092 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls podName:08b38b87-8fd7-475e-94db-8bdd1339dc4b nodeName:}" failed. No retries permitted until 2026-11-02 21:08:30.494086142 +0000 UTC m=+2.933660842 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls") pod "image-registry-588654849f-w89ht" (UID: "08b38b87-8fd7-475e-94db-8bdd1339dc4b") : secret "image-registry-tls" not found Nov 02 21:08:29.494487 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:29.494121 4935 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:08:29.494487 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:29.494157 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls podName:67c3caa0-0b53-42c9-98b2-96addd3dd7fc nodeName:}" failed. No retries permitted until 2026-11-02 21:08:30.494147669 +0000 UTC m=+2.933722419 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2nshl" (UID: "67c3caa0-0b53-42c9-98b2-96addd3dd7fc") : secret "insights-runtime-extractor-tls" not found Nov 02 21:08:29.494487 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:29.494183 4935 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 21:08:29.494487 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:29.494213 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert podName:24c2ea1f-b61b-4e62-914a-2c52508bc04d nodeName:}" failed. No retries permitted until 2026-11-02 21:08:30.494206367 +0000 UTC m=+2.933781067 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qpklq" (UID: "24c2ea1f-b61b-4e62-914a-2c52508bc04d") : secret "networking-console-plugin-cert" not found Nov 02 21:08:29.494487 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:29.494215 4935 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 21:08:29.494487 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:29.494278 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls podName:a899c9c5-79b4-472d-942c-5e0c4c53e954 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:30.494262366 +0000 UTC m=+2.933837080 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-z4kx6" (UID: "a899c9c5-79b4-472d-942c-5e0c4c53e954") : secret "cluster-monitoring-operator-tls" not found Nov 02 21:08:29.602641 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.602604 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:29.829221 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.829191 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/2.log" Nov 02 21:08:29.829646 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.829629 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/1.log" Nov 02 21:08:29.829707 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.829684 4935 generic.go:356] "Generic (PLEG): container finished" podID="9a2820a3-6d1e-42d0-8a89-0e47f870abc4" containerID="1e2b1fa8a7474443c38714e73f31d908b324bff0a47b0290188ad4a0e58bf593" exitCode=255 Nov 02 21:08:29.829805 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.829776 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" event={"ID":"9a2820a3-6d1e-42d0-8a89-0e47f870abc4","Type":"ContainerDied","Data":"1e2b1fa8a7474443c38714e73f31d908b324bff0a47b0290188ad4a0e58bf593"} Nov 02 21:08:29.829855 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.829841 4935 scope.go:122] "RemoveContainer" containerID="25880bea858c09ebae11d9d77d74a3d9d1f8c364288f4a9ac3048abab9b30755" Nov 02 21:08:29.830069 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:29.830052 4935 scope.go:122] "RemoveContainer" containerID="1e2b1fa8a7474443c38714e73f31d908b324bff0a47b0290188ad4a0e58bf593" Nov 02 21:08:29.830290 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:29.830272 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qd9n6_openshift-console-operator(9a2820a3-6d1e-42d0-8a89-0e47f870abc4)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" podUID="9a2820a3-6d1e-42d0-8a89-0e47f870abc4" Nov 02 21:08:30.501463 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:30.501428 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:30.501639 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:30.501471 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-z4kx6\" (UID: \"a899c9c5-79b4-472d-942c-5e0c4c53e954\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6" Nov 02 21:08:30.501639 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:30.501501 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qpklq\" (UID: \"24c2ea1f-b61b-4e62-914a-2c52508bc04d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qpklq" Nov 02 21:08:30.501639 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:30.501524 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:30.501639 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:30.501550 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs\") pod \"network-metrics-daemon-s98xl\" (UID: \"d45d9164-2ba4-400d-8b27-273d7f4f351d\") " pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:08:30.501639 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:30.501577 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vmqkq\" (UID: \"6b5655ce-43ca-48e4-9e4b-872283ccb85c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" Nov 02 21:08:30.501639 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:30.501585 4935 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 21:08:30.501639 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:30.501598 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:30.501901 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:30.501592 4935 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:08:30.501901 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:30.501660 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls podName:a899c9c5-79b4-472d-942c-5e0c4c53e954 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:32.50164065 +0000 UTC m=+4.941215367 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-z4kx6" (UID: "a899c9c5-79b4-472d-942c-5e0c4c53e954") : secret "cluster-monitoring-operator-tls" not found Nov 02 21:08:30.501901 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:30.501664 4935 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-588654849f-w89ht: secret "image-registry-tls" not found Nov 02 21:08:30.501901 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:30.501680 4935 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 21:08:30.501901 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:30.501689 4935 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 21:08:30.501901 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:30.501726 4935 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 21:08:30.501901 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:30.501680 4935 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:08:30.501901 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:30.501687 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle podName:44a209b7-2144-45a0-bea0-2f8adf6658cb nodeName:}" failed. No retries permitted until 2026-11-02 21:08:32.50167683 +0000 UTC m=+4.941251534 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle") pod "router-default-798b7b6b45-hlcl7" (UID: "44a209b7-2144-45a0-bea0-2f8adf6658cb") : configmap references non-existent config key: service-ca.crt Nov 02 21:08:30.501901 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:30.501805 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs podName:d45d9164-2ba4-400d-8b27-273d7f4f351d nodeName:}" failed. No retries permitted until 2026-11-02 21:08:32.501792772 +0000 UTC m=+4.941367475 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs") pod "network-metrics-daemon-s98xl" (UID: "d45d9164-2ba4-400d-8b27-273d7f4f351d") : secret "metrics-daemon-secret" not found Nov 02 21:08:30.501901 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:30.501818 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls podName:08b38b87-8fd7-475e-94db-8bdd1339dc4b nodeName:}" failed. No retries permitted until 2026-11-02 21:08:32.501812705 +0000 UTC m=+4.941387404 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls") pod "image-registry-588654849f-w89ht" (UID: "08b38b87-8fd7-475e-94db-8bdd1339dc4b") : secret "image-registry-tls" not found Nov 02 21:08:30.501901 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:30.501829 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert podName:24c2ea1f-b61b-4e62-914a-2c52508bc04d nodeName:}" failed. No retries permitted until 2026-11-02 21:08:32.50182354 +0000 UTC m=+4.941398240 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qpklq" (UID: "24c2ea1f-b61b-4e62-914a-2c52508bc04d") : secret "networking-console-plugin-cert" not found Nov 02 21:08:30.501901 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:30.501840 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls podName:6b5655ce-43ca-48e4-9e4b-872283ccb85c nodeName:}" failed. No retries permitted until 2026-11-02 21:08:32.501836059 +0000 UTC m=+4.941410758 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vmqkq" (UID: "6b5655ce-43ca-48e4-9e4b-872283ccb85c") : secret "samples-operator-tls" not found Nov 02 21:08:30.501901 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:30.501856 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls podName:67c3caa0-0b53-42c9-98b2-96addd3dd7fc nodeName:}" failed. No retries permitted until 2026-11-02 21:08:32.501852144 +0000 UTC m=+4.941426844 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2nshl" (UID: "67c3caa0-0b53-42c9-98b2-96addd3dd7fc") : secret "insights-runtime-extractor-tls" not found Nov 02 21:08:30.501901 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:30.501898 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:30.502345 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:30.501921 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:30.502345 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:30.501941 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:30.502345 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:30.502007 4935 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:08:30.502345 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:30.502024 4935 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 21:08:30.502345 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:30.502051 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls podName:4ad2a39a-7a74-44a2-a97f-88e84e65616c nodeName:}" failed. No retries permitted until 2026-11-02 21:08:32.502040567 +0000 UTC m=+4.941615272 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls") pod "dns-default-vn62q" (UID: "4ad2a39a-7a74-44a2-a97f-88e84e65616c") : secret "dns-default-metrics-tls" not found Nov 02 21:08:30.502345 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:30.502008 4935 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:08:30.502345 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:30.502064 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs podName:44a209b7-2144-45a0-bea0-2f8adf6658cb nodeName:}" failed. No retries permitted until 2026-11-02 21:08:32.502057961 +0000 UTC m=+4.941632661 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs") pod "router-default-798b7b6b45-hlcl7" (UID: "44a209b7-2144-45a0-bea0-2f8adf6658cb") : secret "router-metrics-certs-default" not found Nov 02 21:08:30.502345 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:30.502064 4935 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c987dc54-2p2h6: secret "image-registry-tls" not found Nov 02 21:08:30.502345 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:30.502090 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls podName:cf9df1a4-3684-464c-97f3-21a3151a7183 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:32.502080026 +0000 UTC m=+4.941654726 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls") pod "image-registry-c987dc54-2p2h6" (UID: "cf9df1a4-3684-464c-97f3-21a3151a7183") : secret "image-registry-tls" not found Nov 02 21:08:30.834081 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:30.834052 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/2.log" Nov 02 21:08:30.834560 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:30.834460 4935 scope.go:122] "RemoveContainer" containerID="1e2b1fa8a7474443c38714e73f31d908b324bff0a47b0290188ad4a0e58bf593" Nov 02 21:08:30.834690 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:30.834666 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qd9n6_openshift-console-operator(9a2820a3-6d1e-42d0-8a89-0e47f870abc4)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" podUID="9a2820a3-6d1e-42d0-8a89-0e47f870abc4" Nov 02 21:08:30.835684 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:30.835663 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wf84d" event={"ID":"39830cb0-c259-446d-b2d3-fd8956b968fb","Type":"ContainerStarted","Data":"ef7c1f54522640636497d5bdf289cfd0dd615034e26acff8f19180fff96f6f37"} Nov 02 21:08:30.908408 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:30.908380 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:31.264988 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:31.264964 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gxdgr_be519c9c-2677-4c4a-99c5-c537317bda3e/dns-node-resolver/0.log" Nov 02 21:08:31.522534 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:31.522468 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:32.014771 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:32.014743 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-77xn9_d49ad8b8-08aa-4da0-9139-b2be1efeb6cd/node-ca/0.log" Nov 02 21:08:32.520076 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:32.520042 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:32.520225 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:32.520096 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:32.520225 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:32.520200 4935 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:08:32.520225 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:32.520212 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:32.520333 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:32.520229 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle podName:44a209b7-2144-45a0-bea0-2f8adf6658cb nodeName:}" failed. No retries permitted until 2026-11-02 21:08:36.520211645 +0000 UTC m=+8.959786360 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle") pod "router-default-798b7b6b45-hlcl7" (UID: "44a209b7-2144-45a0-bea0-2f8adf6658cb") : configmap references non-existent config key: service-ca.crt Nov 02 21:08:32.520333 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:32.520254 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls podName:4ad2a39a-7a74-44a2-a97f-88e84e65616c nodeName:}" failed. No retries permitted until 2026-11-02 21:08:36.520246657 +0000 UTC m=+8.959821356 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls") pod "dns-default-vn62q" (UID: "4ad2a39a-7a74-44a2-a97f-88e84e65616c") : secret "dns-default-metrics-tls" not found Nov 02 21:08:32.520333 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:32.520286 4935 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 21:08:32.520333 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:32.520293 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:32.520333 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:32.520324 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs podName:44a209b7-2144-45a0-bea0-2f8adf6658cb nodeName:}" failed. No retries permitted until 2026-11-02 21:08:36.520314111 +0000 UTC m=+8.959888811 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs") pod "router-default-798b7b6b45-hlcl7" (UID: "44a209b7-2144-45a0-bea0-2f8adf6658cb") : secret "router-metrics-certs-default" not found Nov 02 21:08:32.520505 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:32.520351 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:32.520505 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:32.520363 4935 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:08:32.520505 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:32.520373 4935 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c987dc54-2p2h6: secret "image-registry-tls" not found Nov 02 21:08:32.520505 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:32.520378 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-z4kx6\" (UID: \"a899c9c5-79b4-472d-942c-5e0c4c53e954\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6" Nov 02 21:08:32.520505 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:32.520395 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls podName:cf9df1a4-3684-464c-97f3-21a3151a7183 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:36.520389018 +0000 UTC m=+8.959963754 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls") pod "image-registry-c987dc54-2p2h6" (UID: "cf9df1a4-3684-464c-97f3-21a3151a7183") : secret "image-registry-tls" not found Nov 02 21:08:32.520505 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:32.520433 4935 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 21:08:32.520505 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:32.520425 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qpklq\" (UID: \"24c2ea1f-b61b-4e62-914a-2c52508bc04d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qpklq" Nov 02 21:08:32.520505 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:32.520471 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls podName:a899c9c5-79b4-472d-942c-5e0c4c53e954 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:36.520458729 +0000 UTC m=+8.960033433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-z4kx6" (UID: "a899c9c5-79b4-472d-942c-5e0c4c53e954") : secret "cluster-monitoring-operator-tls" not found Nov 02 21:08:32.520505 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:32.520481 4935 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:08:32.520505 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:32.520496 4935 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-588654849f-w89ht: secret "image-registry-tls" not found Nov 02 21:08:32.520505 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:32.520511 4935 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 21:08:32.520861 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:32.520511 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:32.520861 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:32.520530 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls podName:08b38b87-8fd7-475e-94db-8bdd1339dc4b nodeName:}" failed. No retries permitted until 2026-11-02 21:08:36.520520016 +0000 UTC m=+8.960094730 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls") pod "image-registry-588654849f-w89ht" (UID: "08b38b87-8fd7-475e-94db-8bdd1339dc4b") : secret "image-registry-tls" not found Nov 02 21:08:32.520861 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:32.520551 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert podName:24c2ea1f-b61b-4e62-914a-2c52508bc04d nodeName:}" failed. No retries permitted until 2026-11-02 21:08:36.520541727 +0000 UTC m=+8.960116440 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qpklq" (UID: "24c2ea1f-b61b-4e62-914a-2c52508bc04d") : secret "networking-console-plugin-cert" not found Nov 02 21:08:32.520861 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:32.520567 4935 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:08:32.520861 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:32.520600 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs\") pod \"network-metrics-daemon-s98xl\" (UID: \"d45d9164-2ba4-400d-8b27-273d7f4f351d\") " pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:08:32.520861 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:32.520643 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vmqkq\" (UID: \"6b5655ce-43ca-48e4-9e4b-872283ccb85c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" Nov 02 21:08:32.520861 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:32.520654 4935 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 21:08:32.520861 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:32.520691 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls podName:67c3caa0-0b53-42c9-98b2-96addd3dd7fc nodeName:}" failed. No retries permitted until 2026-11-02 21:08:36.520668243 +0000 UTC m=+8.960242959 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2nshl" (UID: "67c3caa0-0b53-42c9-98b2-96addd3dd7fc") : secret "insights-runtime-extractor-tls" not found Nov 02 21:08:32.520861 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:32.520721 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs podName:d45d9164-2ba4-400d-8b27-273d7f4f351d nodeName:}" failed. No retries permitted until 2026-11-02 21:08:36.520709709 +0000 UTC m=+8.960284416 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs") pod "network-metrics-daemon-s98xl" (UID: "d45d9164-2ba4-400d-8b27-273d7f4f351d") : secret "metrics-daemon-secret" not found Nov 02 21:08:32.520861 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:32.520731 4935 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 21:08:32.520861 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:32.520780 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls podName:6b5655ce-43ca-48e4-9e4b-872283ccb85c nodeName:}" failed. No retries permitted until 2026-11-02 21:08:36.52076901 +0000 UTC m=+8.960343732 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vmqkq" (UID: "6b5655ce-43ca-48e4-9e4b-872283ccb85c") : secret "samples-operator-tls" not found Nov 02 21:08:32.747989 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:32.747959 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:32.815422 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:32.815336 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mlgzw_7de0fc42-d787-49e7-80a2-5122a67bfba0/serve-healthcheck-canary/0.log" Nov 02 21:08:33.174313 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:33.174235 4935 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:33.174673 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:33.174632 4935 scope.go:122] "RemoveContainer" containerID="1e2b1fa8a7474443c38714e73f31d908b324bff0a47b0290188ad4a0e58bf593" Nov 02 21:08:33.174845 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:33.174810 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qd9n6_openshift-console-operator(9a2820a3-6d1e-42d0-8a89-0e47f870abc4)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" podUID="9a2820a3-6d1e-42d0-8a89-0e47f870abc4" Nov 02 21:08:33.211966 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:33.211936 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:33.212177 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:33.212163 4935 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 21:08:33.233527 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:33.233489 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:33.415587 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:33.415541 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-wf84d_39830cb0-c259-446d-b2d3-fd8956b968fb/migrator/0.log" Nov 02 21:08:33.440608 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:33.440535 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:33.618365 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:33.618332 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf_5ca28f2b-c77d-4aa1-8501-ec380253ab88/kube-storage-version-migrator-operator/0.log" Nov 02 21:08:33.734421 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:33.734334 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:33.942144 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:33.942104 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:34.150282 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:34.150250 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:34.529012 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:34.528978 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:34.575325 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:34.575283 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:08:34.919997 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:34.919920 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:35.124677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:35.124646 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:35.322266 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:35.322235 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:35.519752 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:35.519712 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:36.558647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:36.558602 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qpklq\" (UID: \"24c2ea1f-b61b-4e62-914a-2c52508bc04d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qpklq" Nov 02 21:08:36.558647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:36.558655 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:36.559220 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:36.558683 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs\") pod \"network-metrics-daemon-s98xl\" (UID: \"d45d9164-2ba4-400d-8b27-273d7f4f351d\") " pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:08:36.559220 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:36.558707 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vmqkq\" (UID: \"6b5655ce-43ca-48e4-9e4b-872283ccb85c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" Nov 02 21:08:36.559220 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:36.558729 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:36.559220 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:36.558759 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:36.559220 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:36.558759 4935 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 21:08:36.559220 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:36.558778 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:36.559220 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:36.558805 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:36.559220 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:36.558821 4935 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:08:36.559220 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:36.558829 4935 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 21:08:36.559220 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:36.558838 4935 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 21:08:36.559220 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:36.558888 4935 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 21:08:36.559220 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:36.558841 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert podName:24c2ea1f-b61b-4e62-914a-2c52508bc04d nodeName:}" failed. No retries permitted until 2026-11-02 21:08:44.558820824 +0000 UTC m=+16.998395544 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qpklq" (UID: "24c2ea1f-b61b-4e62-914a-2c52508bc04d") : secret "networking-console-plugin-cert" not found Nov 02 21:08:36.559220 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:36.558882 4935 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:08:36.559220 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:36.558922 4935 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c987dc54-2p2h6: secret "image-registry-tls" not found Nov 02 21:08:36.559220 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:36.558932 4935 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:08:36.559697 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:36.558959 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls podName:67c3caa0-0b53-42c9-98b2-96addd3dd7fc nodeName:}" failed. No retries permitted until 2026-11-02 21:08:44.5589301 +0000 UTC m=+16.998504811 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2nshl" (UID: "67c3caa0-0b53-42c9-98b2-96addd3dd7fc") : secret "insights-runtime-extractor-tls" not found Nov 02 21:08:36.559697 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:36.559410 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs podName:d45d9164-2ba4-400d-8b27-273d7f4f351d nodeName:}" failed. No retries permitted until 2026-11-02 21:08:44.55939924 +0000 UTC m=+16.998973941 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs") pod "network-metrics-daemon-s98xl" (UID: "d45d9164-2ba4-400d-8b27-273d7f4f351d") : secret "metrics-daemon-secret" not found Nov 02 21:08:36.559697 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:36.559427 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls podName:6b5655ce-43ca-48e4-9e4b-872283ccb85c nodeName:}" failed. No retries permitted until 2026-11-02 21:08:44.55942102 +0000 UTC m=+16.998995719 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vmqkq" (UID: "6b5655ce-43ca-48e4-9e4b-872283ccb85c") : secret "samples-operator-tls" not found Nov 02 21:08:36.559697 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:36.559443 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs podName:44a209b7-2144-45a0-bea0-2f8adf6658cb nodeName:}" failed. No retries permitted until 2026-11-02 21:08:44.559438171 +0000 UTC m=+16.999012871 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs") pod "router-default-798b7b6b45-hlcl7" (UID: "44a209b7-2144-45a0-bea0-2f8adf6658cb") : secret "router-metrics-certs-default" not found Nov 02 21:08:36.559697 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:36.559458 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls podName:cf9df1a4-3684-464c-97f3-21a3151a7183 nodeName:}" failed. No retries permitted until 2026-11-02 21:08:44.559453033 +0000 UTC m=+16.999027733 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls") pod "image-registry-c987dc54-2p2h6" (UID: "cf9df1a4-3684-464c-97f3-21a3151a7183") : secret "image-registry-tls" not found Nov 02 21:08:36.559697 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:36.559476 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle podName:44a209b7-2144-45a0-bea0-2f8adf6658cb nodeName:}" failed. No retries permitted until 2026-11-02 21:08:44.559469 +0000 UTC m=+16.999043700 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle") pod "router-default-798b7b6b45-hlcl7" (UID: "44a209b7-2144-45a0-bea0-2f8adf6658cb") : configmap references non-existent config key: service-ca.crt Nov 02 21:08:36.559697 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:36.559491 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls podName:4ad2a39a-7a74-44a2-a97f-88e84e65616c nodeName:}" failed. No retries permitted until 2026-11-02 21:08:44.559485493 +0000 UTC m=+16.999060193 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls") pod "dns-default-vn62q" (UID: "4ad2a39a-7a74-44a2-a97f-88e84e65616c") : secret "dns-default-metrics-tls" not found Nov 02 21:08:36.559697 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:36.559520 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:36.559697 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:36.559541 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-z4kx6\" (UID: \"a899c9c5-79b4-472d-942c-5e0c4c53e954\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6" Nov 02 21:08:36.559697 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:36.559634 4935 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 21:08:36.559697 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:36.559648 4935 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-588654849f-w89ht: secret "image-registry-tls" not found Nov 02 21:08:36.559697 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:36.559684 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls podName:08b38b87-8fd7-475e-94db-8bdd1339dc4b nodeName:}" failed. No retries permitted until 2026-11-02 21:08:44.559673283 +0000 UTC m=+16.999248003 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls") pod "image-registry-588654849f-w89ht" (UID: "08b38b87-8fd7-475e-94db-8bdd1339dc4b") : secret "image-registry-tls" not found Nov 02 21:08:36.560089 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:36.559860 4935 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 21:08:36.561929 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:36.561911 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a899c9c5-79b4-472d-942c-5e0c4c53e954-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-z4kx6\" (UID: \"a899c9c5-79b4-472d-942c-5e0c4c53e954\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6" Nov 02 21:08:36.623988 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:36.623958 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6" Nov 02 21:08:36.708621 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:36.708589 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:36.913768 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:36.913682 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:37.031732 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:37.031699 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6"] Nov 02 21:08:37.035512 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:37.035481 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda899c9c5_79b4_472d_942c_5e0c4c53e954.slice/crio-aab45388b043aeecb1712a5641a232b1dea26d0035b44a2b8a5fce2643102122 WatchSource:0}: Error finding container aab45388b043aeecb1712a5641a232b1dea26d0035b44a2b8a5fce2643102122: Status 404 returned error can't find the container with id aab45388b043aeecb1712a5641a232b1dea26d0035b44a2b8a5fce2643102122 Nov 02 21:08:37.123543 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:37.123510 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:37.328409 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:37.328370 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:37.525751 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:37.525710 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:37.858781 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:37.858749 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6" event={"ID":"a899c9c5-79b4-472d-942c-5e0c4c53e954","Type":"ContainerStarted","Data":"aab45388b043aeecb1712a5641a232b1dea26d0035b44a2b8a5fce2643102122"} Nov 02 21:08:38.713247 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:38.713206 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:38.917539 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:38.917501 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:39.114267 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:39.114238 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:39.318738 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:39.318704 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:39.520684 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:39.520645 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:39.602978 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:39.602935 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:39.603379 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:39.603358 4935 scope.go:122] "RemoveContainer" containerID="1e2b1fa8a7474443c38714e73f31d908b324bff0a47b0290188ad4a0e58bf593" Nov 02 21:08:39.718861 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:39.718835 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:39.870110 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:39.870031 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 21:08:39.870505 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:39.870487 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/2.log" Nov 02 21:08:39.870612 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:39.870523 4935 generic.go:356] "Generic (PLEG): container finished" podID="9a2820a3-6d1e-42d0-8a89-0e47f870abc4" containerID="9ee21701eacb3386e530293e8a93a0c5110504a6c55b8ca42ad80bb324f64ec3" exitCode=255 Nov 02 21:08:39.870612 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:39.870599 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" event={"ID":"9a2820a3-6d1e-42d0-8a89-0e47f870abc4","Type":"ContainerDied","Data":"9ee21701eacb3386e530293e8a93a0c5110504a6c55b8ca42ad80bb324f64ec3"} Nov 02 21:08:39.870691 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:39.870680 4935 scope.go:122] "RemoveContainer" containerID="1e2b1fa8a7474443c38714e73f31d908b324bff0a47b0290188ad4a0e58bf593" Nov 02 21:08:39.871059 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:39.871038 4935 scope.go:122] "RemoveContainer" containerID="9ee21701eacb3386e530293e8a93a0c5110504a6c55b8ca42ad80bb324f64ec3" Nov 02 21:08:39.871299 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:39.871280 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qd9n6_openshift-console-operator(9a2820a3-6d1e-42d0-8a89-0e47f870abc4)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" podUID="9a2820a3-6d1e-42d0-8a89-0e47f870abc4" Nov 02 21:08:40.732416 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:40.732373 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:40.875761 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:40.875725 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-z4kx6" event={"ID":"a899c9c5-79b4-472d-942c-5e0c4c53e954","Type":"ContainerStarted","Data":"dc497771a4b01dfbd94d82bc88fd89389450811737ccb1ddab0855621f837e1f"} Nov 02 21:08:40.877230 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:40.877213 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 21:08:40.932580 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:40.932518 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:41.153646 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:41.153614 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:41.370445 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:41.370407 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:41.571477 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:41.571446 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:41.900347 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:41.900315 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:42.096412 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:42.096377 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:42.232546 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:42.232474 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqnmg" Nov 02 21:08:42.306292 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:42.306254 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:42.746424 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:42.746382 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:42.952747 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:42.952712 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:43.148261 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:43.148226 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:43.175073 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:43.175049 4935 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:43.175444 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:43.175428 4935 scope.go:122] "RemoveContainer" containerID="9ee21701eacb3386e530293e8a93a0c5110504a6c55b8ca42ad80bb324f64ec3" Nov 02 21:08:43.175659 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:43.175640 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qd9n6_openshift-console-operator(9a2820a3-6d1e-42d0-8a89-0e47f870abc4)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" podUID="9a2820a3-6d1e-42d0-8a89-0e47f870abc4" Nov 02 21:08:43.352200 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:43.352166 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:43.557924 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:43.557895 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:43.918777 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:43.918702 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:44.637060 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:44.637025 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qpklq\" (UID: \"24c2ea1f-b61b-4e62-914a-2c52508bc04d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qpklq" Nov 02 21:08:44.637542 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:44.637069 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:44.637542 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:44.637099 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs\") pod \"network-metrics-daemon-s98xl\" (UID: \"d45d9164-2ba4-400d-8b27-273d7f4f351d\") " pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:08:44.637542 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:44.637124 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vmqkq\" (UID: \"6b5655ce-43ca-48e4-9e4b-872283ccb85c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" Nov 02 21:08:44.637542 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:44.637261 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:44.637542 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:44.637340 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:44.637542 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:44.637375 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:44.637542 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:44.637411 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:44.637542 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:44.637452 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:44.638088 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:44.638035 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44a209b7-2144-45a0-bea0-2f8adf6658cb-service-ca-bundle\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:44.640311 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:44.640249 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ad2a39a-7a74-44a2-a97f-88e84e65616c-metrics-tls\") pod \"dns-default-vn62q\" (UID: \"4ad2a39a-7a74-44a2-a97f-88e84e65616c\") " pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:44.640431 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:44.640323 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b5655ce-43ca-48e4-9e4b-872283ccb85c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vmqkq\" (UID: \"6b5655ce-43ca-48e4-9e4b-872283ccb85c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" Nov 02 21:08:44.640431 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:44.640353 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67c3caa0-0b53-42c9-98b2-96addd3dd7fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2nshl\" (UID: \"67c3caa0-0b53-42c9-98b2-96addd3dd7fc\") " pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:44.640556 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:44.640465 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45d9164-2ba4-400d-8b27-273d7f4f351d-metrics-certs\") pod \"network-metrics-daemon-s98xl\" (UID: \"d45d9164-2ba4-400d-8b27-273d7f4f351d\") " pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:08:44.640608 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:44.640557 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44a209b7-2144-45a0-bea0-2f8adf6658cb-metrics-certs\") pod \"router-default-798b7b6b45-hlcl7\" (UID: \"44a209b7-2144-45a0-bea0-2f8adf6658cb\") " pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:44.640608 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:44.640572 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls\") pod \"image-registry-588654849f-w89ht\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:44.640679 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:44.640661 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls\") pod \"image-registry-c987dc54-2p2h6\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:44.641004 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:44.640981 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24c2ea1f-b61b-4e62-914a-2c52508bc04d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qpklq\" (UID: \"24c2ea1f-b61b-4e62-914a-2c52508bc04d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qpklq" Nov 02 21:08:44.719292 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:44.719258 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" Nov 02 21:08:44.720369 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:44.720347 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:44.725066 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:44.725041 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:44.725177 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:44.725122 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-qpklq" Nov 02 21:08:44.726550 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:44.726527 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s98xl" Nov 02 21:08:44.727616 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:44.727592 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:44.733713 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:44.733684 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:44.733808 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:44.733786 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2nshl" Nov 02 21:08:45.165244 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:45.165209 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq"] Nov 02 21:08:45.209311 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:45.209281 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c2ea1f_b61b_4e62_914a_2c52508bc04d.slice/crio-dbd26350f6e0ffd77cd39a3be108df4ef79d32a08bbf99e6eb9abfe976d1f5e2 WatchSource:0}: Error finding container dbd26350f6e0ffd77cd39a3be108df4ef79d32a08bbf99e6eb9abfe976d1f5e2: Status 404 returned error can't find the container with id dbd26350f6e0ffd77cd39a3be108df4ef79d32a08bbf99e6eb9abfe976d1f5e2 Nov 02 21:08:45.209620 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:45.209593 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b38b87_8fd7_475e_94db_8bdd1339dc4b.slice/crio-43db9f6992f8e90856affda48a15a37962a6012a0b25924e581d18793824f334 WatchSource:0}: Error finding container 43db9f6992f8e90856affda48a15a37962a6012a0b25924e581d18793824f334: Status 404 returned error can't find the container with id 43db9f6992f8e90856affda48a15a37962a6012a0b25924e581d18793824f334 Nov 02 21:08:45.214645 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:45.212520 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-qpklq"] Nov 02 21:08:45.214645 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:45.213346 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-588654849f-w89ht"] Nov 02 21:08:45.214645 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:45.214553 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd45d9164_2ba4_400d_8b27_273d7f4f351d.slice/crio-95d89152aebde69c2971767177b30b1e211d808e968df8277f730272edd1e622 WatchSource:0}: Error finding container 95d89152aebde69c2971767177b30b1e211d808e968df8277f730272edd1e622: Status 404 returned error can't find the container with id 95d89152aebde69c2971767177b30b1e211d808e968df8277f730272edd1e622 Nov 02 21:08:45.225449 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:45.225427 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a209b7_2144_45a0_bea0_2f8adf6658cb.slice/crio-a1547f4444d32cda4ed15e1d26a952e2d0b486848c36d8f6e52f16a7e759db05 WatchSource:0}: Error finding container a1547f4444d32cda4ed15e1d26a952e2d0b486848c36d8f6e52f16a7e759db05: Status 404 returned error can't find the container with id a1547f4444d32cda4ed15e1d26a952e2d0b486848c36d8f6e52f16a7e759db05 Nov 02 21:08:45.250034 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:45.250008 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s98xl"] Nov 02 21:08:45.260691 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:45.260669 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-798b7b6b45-hlcl7"] Nov 02 21:08:45.261543 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:45.261333 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c3caa0_0b53_42c9_98b2_96addd3dd7fc.slice/crio-538ca0681f984b51349dbc004132bdc270c330285f36d7b82aada01a2c9c939e WatchSource:0}: Error finding container 538ca0681f984b51349dbc004132bdc270c330285f36d7b82aada01a2c9c939e: Status 404 returned error can't find the container with id 538ca0681f984b51349dbc004132bdc270c330285f36d7b82aada01a2c9c939e Nov 02 21:08:45.262107 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:45.262088 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2nshl"] Nov 02 21:08:45.262507 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:45.262342 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vn62q"] Nov 02 21:08:45.263307 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:45.263086 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c987dc54-2p2h6"] Nov 02 21:08:45.265868 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:45.265845 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf9df1a4_3684_464c_97f3_21a3151a7183.slice/crio-62e18e11b74c4ea37ad7531f1f14f74da34f8d98ea2f483d9a7343d100f0cc07 WatchSource:0}: Error finding container 62e18e11b74c4ea37ad7531f1f14f74da34f8d98ea2f483d9a7343d100f0cc07: Status 404 returned error can't find the container with id 62e18e11b74c4ea37ad7531f1f14f74da34f8d98ea2f483d9a7343d100f0cc07 Nov 02 21:08:45.895275 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:45.895236 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s98xl" event={"ID":"d45d9164-2ba4-400d-8b27-273d7f4f351d","Type":"ContainerStarted","Data":"95d89152aebde69c2971767177b30b1e211d808e968df8277f730272edd1e622"} Nov 02 21:08:45.896529 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:45.896499 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" event={"ID":"6b5655ce-43ca-48e4-9e4b-872283ccb85c","Type":"ContainerStarted","Data":"42cfd842f9f231351254d603b8970f87e44faa5e8ebb039ff40df104b7f3676a"} Nov 02 21:08:45.897995 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:45.897971 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2nshl" event={"ID":"67c3caa0-0b53-42c9-98b2-96addd3dd7fc","Type":"ContainerStarted","Data":"22b5c713ff73b7d4b0b193d36853e4bc56424c819721c97a5a31adcb147b066a"} Nov 02 21:08:45.898096 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:45.898020 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2nshl" event={"ID":"67c3caa0-0b53-42c9-98b2-96addd3dd7fc","Type":"ContainerStarted","Data":"538ca0681f984b51349dbc004132bdc270c330285f36d7b82aada01a2c9c939e"} Nov 02 21:08:45.899518 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:45.899494 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c987dc54-2p2h6" event={"ID":"cf9df1a4-3684-464c-97f3-21a3151a7183","Type":"ContainerStarted","Data":"4c2ae0c2441064fac6a2e8a2550c8f267fac138ceb60f8ff6e473aa7337c8328"} Nov 02 21:08:45.899606 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:45.899534 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c987dc54-2p2h6" event={"ID":"cf9df1a4-3684-464c-97f3-21a3151a7183","Type":"ContainerStarted","Data":"62e18e11b74c4ea37ad7531f1f14f74da34f8d98ea2f483d9a7343d100f0cc07"} Nov 02 21:08:45.899606 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:45.899586 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:08:45.901080 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:45.901061 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-588654849f-w89ht" event={"ID":"08b38b87-8fd7-475e-94db-8bdd1339dc4b","Type":"ContainerStarted","Data":"c28d18052aa31e1bb5a4f5a62ae4b32f6960f10597af7d3fbf9b0fda50d8b0ad"} Nov 02 21:08:45.901161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:45.901093 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-588654849f-w89ht" event={"ID":"08b38b87-8fd7-475e-94db-8bdd1339dc4b","Type":"ContainerStarted","Data":"43db9f6992f8e90856affda48a15a37962a6012a0b25924e581d18793824f334"} Nov 02 21:08:45.901213 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:45.901167 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:08:45.902237 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:45.902219 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vn62q" event={"ID":"4ad2a39a-7a74-44a2-a97f-88e84e65616c","Type":"ContainerStarted","Data":"019e17660394cb9dec6b01178808c9097ca5d94b71ed5fcacaa219efb06542ba"} Nov 02 21:08:45.903528 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:45.903501 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-798b7b6b45-hlcl7" event={"ID":"44a209b7-2144-45a0-bea0-2f8adf6658cb","Type":"ContainerStarted","Data":"7212fcc442126ef1411d09d849b4dbbd02748ce149492fd6b615e98fefeec3cc"} Nov 02 21:08:45.903600 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:45.903535 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-798b7b6b45-hlcl7" event={"ID":"44a209b7-2144-45a0-bea0-2f8adf6658cb","Type":"ContainerStarted","Data":"a1547f4444d32cda4ed15e1d26a952e2d0b486848c36d8f6e52f16a7e759db05"} Nov 02 21:08:45.904588 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:45.904566 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-qpklq" event={"ID":"24c2ea1f-b61b-4e62-914a-2c52508bc04d","Type":"ContainerStarted","Data":"dbd26350f6e0ffd77cd39a3be108df4ef79d32a08bbf99e6eb9abfe976d1f5e2"} Nov 02 21:08:46.005593 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:46.005559 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-588654849f-w89ht"] Nov 02 21:08:46.255503 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:46.255426 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:46.464695 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:46.464613 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:46.690474 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:46.690437 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:46.726050 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:46.726025 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:46.729059 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:46.729034 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:46.894995 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:46.894962 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:46.915917 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:46.915841 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2nshl" event={"ID":"67c3caa0-0b53-42c9-98b2-96addd3dd7fc","Type":"ContainerStarted","Data":"212b1901953d3311fa087b994b2096b864a48053a13701e8c5464b95c22ab72c"} Nov 02 21:08:46.916956 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:46.916928 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:46.918362 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:46.918332 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-798b7b6b45-hlcl7" Nov 02 21:08:47.537342 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:47.537312 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:47.740114 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:47.740050 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:47.923865 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:47.923826 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vn62q" event={"ID":"4ad2a39a-7a74-44a2-a97f-88e84e65616c","Type":"ContainerStarted","Data":"a2d95c55262207822cb35872fdbe7fd6bab0355aa2e42e77be796c36327cf367"} Nov 02 21:08:47.924341 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:47.923889 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vn62q" event={"ID":"4ad2a39a-7a74-44a2-a97f-88e84e65616c","Type":"ContainerStarted","Data":"b096dbc68619b1e445692d2ee8fb549fded1d5810eec81c67635863e455a185d"} Nov 02 21:08:47.924341 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:47.923949 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:47.925735 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:47.925706 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-qpklq" event={"ID":"24c2ea1f-b61b-4e62-914a-2c52508bc04d","Type":"ContainerStarted","Data":"1fc714fdd9a40e9c343eeb0f9435c82083c5731231f9df252d4398583c359d85"} Nov 02 21:08:47.933074 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:47.933046 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s98xl" event={"ID":"d45d9164-2ba4-400d-8b27-273d7f4f351d","Type":"ContainerStarted","Data":"9faed41a1f6c38960da7f04ecefe663d89bc61918338f5966fde62c6843f5b04"} Nov 02 21:08:47.933231 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:47.933086 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s98xl" event={"ID":"d45d9164-2ba4-400d-8b27-273d7f4f351d","Type":"ContainerStarted","Data":"6a1d4a6090bab238cfc44b34585662052788273f59f6d3bcfb70b423fa102a0b"} Nov 02 21:08:47.951232 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:47.951205 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:48.156761 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:48.156684 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:48.933547 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:48.933515 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:48.938489 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:48.938461 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" event={"ID":"6b5655ce-43ca-48e4-9e4b-872283ccb85c","Type":"ContainerStarted","Data":"118dd1858c3b9a6ff6585020e202ee364854a53541ad94f549b36b2c6124ba44"} Nov 02 21:08:48.938624 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:48.938551 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmqkq" event={"ID":"6b5655ce-43ca-48e4-9e4b-872283ccb85c","Type":"ContainerStarted","Data":"a6fba9bdec7a5b80b56be0d906e746a4932c70568ab6ddb1b8f8abf6afa37897"} Nov 02 21:08:48.940792 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:48.940768 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2nshl" event={"ID":"67c3caa0-0b53-42c9-98b2-96addd3dd7fc","Type":"ContainerStarted","Data":"2e9a1b5e4a2a62429178e346719d15afbec884a4fd385163da3298833ecd751b"} Nov 02 21:08:49.328881 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:49.328843 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:49.532681 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:49.532648 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:49.602917 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:49.602837 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:08:49.603320 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:49.603305 4935 scope.go:122] "RemoveContainer" containerID="9ee21701eacb3386e530293e8a93a0c5110504a6c55b8ca42ad80bb324f64ec3" Nov 02 21:08:49.603513 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:49.603495 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-qd9n6_openshift-console-operator(9a2820a3-6d1e-42d0-8a89-0e47f870abc4)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" podUID="9a2820a3-6d1e-42d0-8a89-0e47f870abc4" Nov 02 21:08:50.171795 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:50.171745 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:50.382748 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:50.382701 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:50.588567 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:50.588523 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:50.783559 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:50.783529 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:52.367251 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:52.367217 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:52.568629 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:52.568592 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:54.578640 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:54.578604 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8c67p" Nov 02 21:08:55.790721 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:55.790691 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:56.466749 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:56.466709 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:56.682978 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:56.682938 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:56.901652 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:56.901619 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:57.103549 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.103512 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:57.165792 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.165711 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-86qhj"] Nov 02 21:08:57.203888 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.203859 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.246723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.246687 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n8zf\" (UniqueName: \"kubernetes.io/projected/2dbc0749-14de-4ad6-bc67-f593966c5a1f-kube-api-access-7n8zf\") pod \"node-exporter-86qhj\" (UID: \"2dbc0749-14de-4ad6-bc67-f593966c5a1f\") " pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.246890 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.246730 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2dbc0749-14de-4ad6-bc67-f593966c5a1f-metrics-client-ca\") pod \"node-exporter-86qhj\" (UID: \"2dbc0749-14de-4ad6-bc67-f593966c5a1f\") " pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.246890 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.246752 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2dbc0749-14de-4ad6-bc67-f593966c5a1f-root\") pod \"node-exporter-86qhj\" (UID: \"2dbc0749-14de-4ad6-bc67-f593966c5a1f\") " pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.246890 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.246875 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2dbc0749-14de-4ad6-bc67-f593966c5a1f-sys\") pod \"node-exporter-86qhj\" (UID: \"2dbc0749-14de-4ad6-bc67-f593966c5a1f\") " pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.247037 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.246913 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2dbc0749-14de-4ad6-bc67-f593966c5a1f-node-exporter-tls\") pod \"node-exporter-86qhj\" (UID: \"2dbc0749-14de-4ad6-bc67-f593966c5a1f\") " pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.247037 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.246988 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2dbc0749-14de-4ad6-bc67-f593966c5a1f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-86qhj\" (UID: \"2dbc0749-14de-4ad6-bc67-f593966c5a1f\") " pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.247117 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.247054 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2dbc0749-14de-4ad6-bc67-f593966c5a1f-node-exporter-textfile\") pod \"node-exporter-86qhj\" (UID: \"2dbc0749-14de-4ad6-bc67-f593966c5a1f\") " pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.247117 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.247090 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2dbc0749-14de-4ad6-bc67-f593966c5a1f-node-exporter-wtmp\") pod \"node-exporter-86qhj\" (UID: \"2dbc0749-14de-4ad6-bc67-f593966c5a1f\") " pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.247223 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.247115 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2dbc0749-14de-4ad6-bc67-f593966c5a1f-node-exporter-accelerators-collector-config\") pod \"node-exporter-86qhj\" (UID: \"2dbc0749-14de-4ad6-bc67-f593966c5a1f\") " pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.308638 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.308600 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 21:08:57.308638 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.308611 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 21:08:57.308638 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.308638 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-b7xvl\"" Nov 02 21:08:57.308867 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.308671 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 21:08:57.308867 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.308699 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 21:08:57.313388 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.313366 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:57.347748 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.347720 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2dbc0749-14de-4ad6-bc67-f593966c5a1f-node-exporter-accelerators-collector-config\") pod \"node-exporter-86qhj\" (UID: \"2dbc0749-14de-4ad6-bc67-f593966c5a1f\") " pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.347891 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.347771 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n8zf\" (UniqueName: \"kubernetes.io/projected/2dbc0749-14de-4ad6-bc67-f593966c5a1f-kube-api-access-7n8zf\") pod \"node-exporter-86qhj\" (UID: \"2dbc0749-14de-4ad6-bc67-f593966c5a1f\") " pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.347891 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.347804 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2dbc0749-14de-4ad6-bc67-f593966c5a1f-metrics-client-ca\") pod \"node-exporter-86qhj\" (UID: \"2dbc0749-14de-4ad6-bc67-f593966c5a1f\") " pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.347891 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.347838 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2dbc0749-14de-4ad6-bc67-f593966c5a1f-root\") pod \"node-exporter-86qhj\" (UID: \"2dbc0749-14de-4ad6-bc67-f593966c5a1f\") " pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.347891 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.347863 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2dbc0749-14de-4ad6-bc67-f593966c5a1f-sys\") pod \"node-exporter-86qhj\" (UID: \"2dbc0749-14de-4ad6-bc67-f593966c5a1f\") " pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.347891 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.347887 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2dbc0749-14de-4ad6-bc67-f593966c5a1f-node-exporter-tls\") pod \"node-exporter-86qhj\" (UID: \"2dbc0749-14de-4ad6-bc67-f593966c5a1f\") " pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.348192 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.347920 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2dbc0749-14de-4ad6-bc67-f593966c5a1f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-86qhj\" (UID: \"2dbc0749-14de-4ad6-bc67-f593966c5a1f\") " pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.348192 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.347957 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2dbc0749-14de-4ad6-bc67-f593966c5a1f-node-exporter-textfile\") pod \"node-exporter-86qhj\" (UID: \"2dbc0749-14de-4ad6-bc67-f593966c5a1f\") " pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.348192 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.347991 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2dbc0749-14de-4ad6-bc67-f593966c5a1f-node-exporter-wtmp\") pod \"node-exporter-86qhj\" (UID: \"2dbc0749-14de-4ad6-bc67-f593966c5a1f\") " pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.348192 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.347995 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2dbc0749-14de-4ad6-bc67-f593966c5a1f-root\") pod \"node-exporter-86qhj\" (UID: \"2dbc0749-14de-4ad6-bc67-f593966c5a1f\") " pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.348192 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.348174 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2dbc0749-14de-4ad6-bc67-f593966c5a1f-node-exporter-wtmp\") pod \"node-exporter-86qhj\" (UID: \"2dbc0749-14de-4ad6-bc67-f593966c5a1f\") " pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.348447 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.348231 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2dbc0749-14de-4ad6-bc67-f593966c5a1f-sys\") pod \"node-exporter-86qhj\" (UID: \"2dbc0749-14de-4ad6-bc67-f593966c5a1f\") " pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.348447 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.348344 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2dbc0749-14de-4ad6-bc67-f593966c5a1f-node-exporter-textfile\") pod \"node-exporter-86qhj\" (UID: \"2dbc0749-14de-4ad6-bc67-f593966c5a1f\") " pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.348561 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.348469 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2dbc0749-14de-4ad6-bc67-f593966c5a1f-node-exporter-accelerators-collector-config\") pod \"node-exporter-86qhj\" (UID: \"2dbc0749-14de-4ad6-bc67-f593966c5a1f\") " pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.348694 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.348674 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2dbc0749-14de-4ad6-bc67-f593966c5a1f-metrics-client-ca\") pod \"node-exporter-86qhj\" (UID: \"2dbc0749-14de-4ad6-bc67-f593966c5a1f\") " pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.350646 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.350621 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2dbc0749-14de-4ad6-bc67-f593966c5a1f-node-exporter-tls\") pod \"node-exporter-86qhj\" (UID: \"2dbc0749-14de-4ad6-bc67-f593966c5a1f\") " pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.350646 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.350644 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2dbc0749-14de-4ad6-bc67-f593966c5a1f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-86qhj\" (UID: \"2dbc0749-14de-4ad6-bc67-f593966c5a1f\") " pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.402924 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.402890 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n8zf\" (UniqueName: \"kubernetes.io/projected/2dbc0749-14de-4ad6-bc67-f593966c5a1f-kube-api-access-7n8zf\") pod \"node-exporter-86qhj\" (UID: \"2dbc0749-14de-4ad6-bc67-f593966c5a1f\") " pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.514324 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.514293 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:57.554285 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.554250 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-86qhj" Nov 02 21:08:57.593338 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:08:57.593306 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dbc0749_14de_4ad6_bc67_f593966c5a1f.slice/crio-e03e797eef336b51ef5de453430feaac47b59ac32ef8e5fef245166e17007e87 WatchSource:0}: Error finding container e03e797eef336b51ef5de453430feaac47b59ac32ef8e5fef245166e17007e87: Status 404 returned error can't find the container with id e03e797eef336b51ef5de453430feaac47b59ac32ef8e5fef245166e17007e87 Nov 02 21:08:57.858163 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.858064 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:57.884360 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.884333 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mz5qm" Nov 02 21:08:57.943072 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.943044 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vn62q" Nov 02 21:08:57.975561 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:57.975532 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-86qhj" event={"ID":"2dbc0749-14de-4ad6-bc67-f593966c5a1f","Type":"ContainerStarted","Data":"e03e797eef336b51ef5de453430feaac47b59ac32ef8e5fef245166e17007e87"} Nov 02 21:08:58.468443 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:08:58.468402 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:08:58.548363 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:58.548326 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:58.751819 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:58.751739 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:58.954686 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:58.954655 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:59.153298 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:59.153264 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:08:59.395407 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:08:59.395377 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:00.987792 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:00.987758 4935 generic.go:356] "Generic (PLEG): container finished" podID="2dbc0749-14de-4ad6-bc67-f593966c5a1f" containerID="bb2f97da4bb2e087ae4dd674bb4742a84ee003012e945365a6674eff49e33ce9" exitCode=0 Nov 02 21:09:00.988298 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:00.987851 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-86qhj" event={"ID":"2dbc0749-14de-4ad6-bc67-f593966c5a1f","Type":"ContainerDied","Data":"bb2f97da4bb2e087ae4dd674bb4742a84ee003012e945365a6674eff49e33ce9"} Nov 02 21:09:01.101395 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:01.101366 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:01.307226 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:01.307190 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:01.509292 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:01.509264 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:01.886145 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:01.886095 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:01.994386 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:01.994352 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-86qhj" event={"ID":"2dbc0749-14de-4ad6-bc67-f593966c5a1f","Type":"ContainerStarted","Data":"db99156673d783b5c839f658734170bc185b5c5d09632284f6c4b78d389743e2"} Nov 02 21:09:01.994772 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:01.994409 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-86qhj" event={"ID":"2dbc0749-14de-4ad6-bc67-f593966c5a1f","Type":"ContainerStarted","Data":"5904a021ef447384822d83e36c32e486c37b362cd43cc50df19aa6cc80dfecf4"} Nov 02 21:09:02.081287 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:02.081253 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:02.104245 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:02.104183 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-86qhj" podStartSLOduration=2.713302917 podStartE2EDuration="5.104165946s" totalImagesPullingTime="2.312602425s" totalInitContainerRuntime="78.260604ms" isStatefulPod=true podCreationTimestamp="2026-11-02 21:08:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:09:02.103924376 +0000 UTC m=+34.543499098" watchObservedRunningTime="2026-11-02 21:09:02.104165946 +0000 UTC m=+34.543740688" Nov 02 21:09:02.800579 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:02.800545 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:03.032589 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:03.032553 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:03.769315 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:03.769287 4935 scope.go:122] "RemoveContainer" containerID="9ee21701eacb3386e530293e8a93a0c5110504a6c55b8ca42ad80bb324f64ec3" Nov 02 21:09:03.808376 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:03.808350 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:04.004414 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:04.004382 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 21:09:04.004594 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:04.004444 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" event={"ID":"9a2820a3-6d1e-42d0-8a89-0e47f870abc4","Type":"ContainerStarted","Data":"e1f73040942366806d7e9cdbfad893968706e52c5afef39dbac4117eb891aad8"} Nov 02 21:09:04.004716 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:04.004695 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:09:04.005932 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:04.005895 4935 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-qd9n6 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.14:8443/readyz\": dial tcp 10.132.0.14:8443: connect: connection refused" start-of-body= Nov 02 21:09:04.006043 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:04.005946 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" podUID="9a2820a3-6d1e-42d0-8a89-0e47f870abc4" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.14:8443/readyz\": dial tcp 10.132.0.14:8443: connect: connection refused" Nov 02 21:09:04.015146 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:04.015108 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:05.012170 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:05.012126 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-qd9n6" Nov 02 21:09:05.309607 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:05.309529 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:05.520831 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:05.520800 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:05.721364 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:05.721282 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:05.923493 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:05.923456 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:06.133108 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:06.133077 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:06.920535 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:06.920507 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:09:07.381578 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:07.381547 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:07.612314 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:07.612278 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:07.818717 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:07.818685 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:07.939255 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:07.939227 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:09:08.176299 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:08.176191 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c987dc54-2p2h6"] Nov 02 21:09:08.340395 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:08.340358 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:08.549269 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:08.549239 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:09.330717 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:09.330683 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:12.598174 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:12.598117 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:12.800911 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:12.800876 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:12.960281 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:12.960183 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-588654849f-w89ht" podUID="08b38b87-8fd7-475e-94db-8bdd1339dc4b" containerName="registry" containerID="cri-o://c28d18052aa31e1bb5a4f5a62ae4b32f6960f10597af7d3fbf9b0fda50d8b0ad" gracePeriod=30 Nov 02 21:09:13.003553 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:13.003527 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:13.185734 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:13.185710 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:09:13.285679 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:13.285649 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-kube-api-access-bxbh5\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-kube-api-access-bxbh5\") pod \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " Nov 02 21:09:13.285852 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:13.285710 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08b38b87-8fd7-475e-94db-8bdd1339dc4b-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08b38b87-8fd7-475e-94db-8bdd1339dc4b-installation-pull-secrets\") pod \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " Nov 02 21:09:13.285852 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:13.285734 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08b38b87-8fd7-475e-94db-8bdd1339dc4b-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08b38b87-8fd7-475e-94db-8bdd1339dc4b-ca-trust-extracted\") pod \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " Nov 02 21:09:13.285852 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:13.285771 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-bound-sa-token\") pod \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " Nov 02 21:09:13.285852 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:13.285793 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08b38b87-8fd7-475e-94db-8bdd1339dc4b-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08b38b87-8fd7-475e-94db-8bdd1339dc4b-image-registry-private-configuration\") pod \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " Nov 02 21:09:13.285852 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:13.285815 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-certificates\") pod \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " Nov 02 21:09:13.286102 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:13.285888 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls\") pod \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " Nov 02 21:09:13.286102 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:13.285934 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08b38b87-8fd7-475e-94db-8bdd1339dc4b-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08b38b87-8fd7-475e-94db-8bdd1339dc4b-trusted-ca\") pod \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\" (UID: \"08b38b87-8fd7-475e-94db-8bdd1339dc4b\") " Nov 02 21:09:13.286585 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:13.286510 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-certificates" pod "08b38b87-8fd7-475e-94db-8bdd1339dc4b" (UID: "08b38b87-8fd7-475e-94db-8bdd1339dc4b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:09:13.286585 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:13.286539 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b38b87-8fd7-475e-94db-8bdd1339dc4b-trusted-ca" pod "08b38b87-8fd7-475e-94db-8bdd1339dc4b" (UID: "08b38b87-8fd7-475e-94db-8bdd1339dc4b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:09:13.288840 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:13.288810 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08b38b87-8fd7-475e-94db-8bdd1339dc4b-installation-pull-secrets" pod "08b38b87-8fd7-475e-94db-8bdd1339dc4b" (UID: "08b38b87-8fd7-475e-94db-8bdd1339dc4b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:09:13.288944 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:13.288853 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-bound-sa-token" pod "08b38b87-8fd7-475e-94db-8bdd1339dc4b" (UID: "08b38b87-8fd7-475e-94db-8bdd1339dc4b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:09:13.288944 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:13.288879 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls" pod "08b38b87-8fd7-475e-94db-8bdd1339dc4b" (UID: "08b38b87-8fd7-475e-94db-8bdd1339dc4b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:09:13.288944 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:13.288892 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08b38b87-8fd7-475e-94db-8bdd1339dc4b-image-registry-private-configuration" pod "08b38b87-8fd7-475e-94db-8bdd1339dc4b" (UID: "08b38b87-8fd7-475e-94db-8bdd1339dc4b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:09:13.289059 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:13.288954 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-kube-api-access-bxbh5" pod "08b38b87-8fd7-475e-94db-8bdd1339dc4b" (UID: "08b38b87-8fd7-475e-94db-8bdd1339dc4b"). InnerVolumeSpecName "kube-api-access-bxbh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:09:13.292954 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:13.292853 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b38b87-8fd7-475e-94db-8bdd1339dc4b-ca-trust-extracted" pod "08b38b87-8fd7-475e-94db-8bdd1339dc4b" (UID: "08b38b87-8fd7-475e-94db-8bdd1339dc4b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:09:13.387073 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:13.387037 4935 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:13.387073 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:13.387069 4935 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08b38b87-8fd7-475e-94db-8bdd1339dc4b-trusted-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:13.387296 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:13.387086 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxbh5\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-kube-api-access-bxbh5\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:13.387296 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:13.387101 4935 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08b38b87-8fd7-475e-94db-8bdd1339dc4b-installation-pull-secrets\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:13.387296 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:13.387114 4935 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08b38b87-8fd7-475e-94db-8bdd1339dc4b-ca-trust-extracted\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:13.387296 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:13.387125 4935 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08b38b87-8fd7-475e-94db-8bdd1339dc4b-bound-sa-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:13.387296 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:13.387160 4935 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08b38b87-8fd7-475e-94db-8bdd1339dc4b-image-registry-private-configuration\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:13.387296 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:13.387175 4935 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08b38b87-8fd7-475e-94db-8bdd1339dc4b-registry-certificates\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:13.785807 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:13.785762 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:13.983643 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:13.983611 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:14.043826 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:14.043744 4935 generic.go:356] "Generic (PLEG): container finished" podID="08b38b87-8fd7-475e-94db-8bdd1339dc4b" containerID="c28d18052aa31e1bb5a4f5a62ae4b32f6960f10597af7d3fbf9b0fda50d8b0ad" exitCode=0 Nov 02 21:09:14.043826 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:14.043794 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-588654849f-w89ht" Nov 02 21:09:14.043826 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:14.043802 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-588654849f-w89ht" event={"ID":"08b38b87-8fd7-475e-94db-8bdd1339dc4b","Type":"ContainerDied","Data":"c28d18052aa31e1bb5a4f5a62ae4b32f6960f10597af7d3fbf9b0fda50d8b0ad"} Nov 02 21:09:14.044053 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:14.043846 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-588654849f-w89ht" event={"ID":"08b38b87-8fd7-475e-94db-8bdd1339dc4b","Type":"ContainerDied","Data":"43db9f6992f8e90856affda48a15a37962a6012a0b25924e581d18793824f334"} Nov 02 21:09:14.044053 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:14.043867 4935 scope.go:122] "RemoveContainer" containerID="c28d18052aa31e1bb5a4f5a62ae4b32f6960f10597af7d3fbf9b0fda50d8b0ad" Nov 02 21:09:14.052123 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:14.052104 4935 scope.go:122] "RemoveContainer" containerID="c28d18052aa31e1bb5a4f5a62ae4b32f6960f10597af7d3fbf9b0fda50d8b0ad" Nov 02 21:09:14.052380 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:09:14.052359 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c28d18052aa31e1bb5a4f5a62ae4b32f6960f10597af7d3fbf9b0fda50d8b0ad\": container with ID starting with c28d18052aa31e1bb5a4f5a62ae4b32f6960f10597af7d3fbf9b0fda50d8b0ad not found: ID does not exist" containerID="c28d18052aa31e1bb5a4f5a62ae4b32f6960f10597af7d3fbf9b0fda50d8b0ad" Nov 02 21:09:14.052434 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:14.052389 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c28d18052aa31e1bb5a4f5a62ae4b32f6960f10597af7d3fbf9b0fda50d8b0ad"} err="failed to get container status \"c28d18052aa31e1bb5a4f5a62ae4b32f6960f10597af7d3fbf9b0fda50d8b0ad\": rpc error: code = NotFound desc = could not find container \"c28d18052aa31e1bb5a4f5a62ae4b32f6960f10597af7d3fbf9b0fda50d8b0ad\": container with ID starting with c28d18052aa31e1bb5a4f5a62ae4b32f6960f10597af7d3fbf9b0fda50d8b0ad not found: ID does not exist" Nov 02 21:09:14.243740 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:14.243706 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-588654849f-w89ht"] Nov 02 21:09:14.290055 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:14.290021 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-588654849f-w89ht"] Nov 02 21:09:15.774180 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:15.774121 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08b38b87-8fd7-475e-94db-8bdd1339dc4b" path="/var/lib/kubelet/pods/08b38b87-8fd7-475e-94db-8bdd1339dc4b/volumes" Nov 02 21:09:16.239425 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:16.239349 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:16.446991 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:16.446963 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:16.918182 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:16.918151 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:17.118822 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:17.118791 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:17.647821 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:17.647780 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:17.878733 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:17.878699 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:24.393528 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:24.393492 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:28.468075 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:09:28.468029 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:09:28.969605 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:28.969572 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:29.175709 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:29.175668 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:29.594492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:29.594452 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:30.100832 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:30.100790 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:30.350363 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:30.350329 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:33.196310 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:33.196248 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-c987dc54-2p2h6" podUID="cf9df1a4-3684-464c-97f3-21a3151a7183" containerName="registry" containerID="cri-o://4c2ae0c2441064fac6a2e8a2550c8f267fac138ceb60f8ff6e473aa7337c8328" gracePeriod=30 Nov 02 21:09:33.443334 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:33.443309 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:09:33.552103 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:33.552072 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf9df1a4-3684-464c-97f3-21a3151a7183-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cf9df1a4-3684-464c-97f3-21a3151a7183-image-registry-private-configuration\") pod \"cf9df1a4-3684-464c-97f3-21a3151a7183\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " Nov 02 21:09:33.552293 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:33.552111 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-bound-sa-token\") pod \"cf9df1a4-3684-464c-97f3-21a3151a7183\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " Nov 02 21:09:33.552293 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:33.552159 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf9df1a4-3684-464c-97f3-21a3151a7183-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cf9df1a4-3684-464c-97f3-21a3151a7183-installation-pull-secrets\") pod \"cf9df1a4-3684-464c-97f3-21a3151a7183\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " Nov 02 21:09:33.552293 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:33.552180 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls\") pod \"cf9df1a4-3684-464c-97f3-21a3151a7183\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " Nov 02 21:09:33.552293 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:33.552214 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf9df1a4-3684-464c-97f3-21a3151a7183-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cf9df1a4-3684-464c-97f3-21a3151a7183-trusted-ca\") pod \"cf9df1a4-3684-464c-97f3-21a3151a7183\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " Nov 02 21:09:33.552293 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:33.552248 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-kube-api-access-kgxs2\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-kube-api-access-kgxs2\") pod \"cf9df1a4-3684-464c-97f3-21a3151a7183\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " Nov 02 21:09:33.552293 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:33.552283 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-certificates\") pod \"cf9df1a4-3684-464c-97f3-21a3151a7183\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " Nov 02 21:09:33.552645 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:33.552331 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9df1a4-3684-464c-97f3-21a3151a7183-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cf9df1a4-3684-464c-97f3-21a3151a7183-ca-trust-extracted\") pod \"cf9df1a4-3684-464c-97f3-21a3151a7183\" (UID: \"cf9df1a4-3684-464c-97f3-21a3151a7183\") " Nov 02 21:09:33.552833 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:33.552766 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf9df1a4-3684-464c-97f3-21a3151a7183-trusted-ca" pod "cf9df1a4-3684-464c-97f3-21a3151a7183" (UID: "cf9df1a4-3684-464c-97f3-21a3151a7183"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:09:33.553035 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:33.553003 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-certificates" pod "cf9df1a4-3684-464c-97f3-21a3151a7183" (UID: "cf9df1a4-3684-464c-97f3-21a3151a7183"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:09:33.554803 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:33.554775 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf9df1a4-3684-464c-97f3-21a3151a7183-installation-pull-secrets" pod "cf9df1a4-3684-464c-97f3-21a3151a7183" (UID: "cf9df1a4-3684-464c-97f3-21a3151a7183"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:09:33.555038 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:33.555015 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls" pod "cf9df1a4-3684-464c-97f3-21a3151a7183" (UID: "cf9df1a4-3684-464c-97f3-21a3151a7183"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:09:33.555112 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:33.555048 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-kube-api-access-kgxs2" pod "cf9df1a4-3684-464c-97f3-21a3151a7183" (UID: "cf9df1a4-3684-464c-97f3-21a3151a7183"). InnerVolumeSpecName "kube-api-access-kgxs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:09:33.555179 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:33.555148 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-bound-sa-token" pod "cf9df1a4-3684-464c-97f3-21a3151a7183" (UID: "cf9df1a4-3684-464c-97f3-21a3151a7183"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:09:33.555215 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:33.555179 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf9df1a4-3684-464c-97f3-21a3151a7183-image-registry-private-configuration" pod "cf9df1a4-3684-464c-97f3-21a3151a7183" (UID: "cf9df1a4-3684-464c-97f3-21a3151a7183"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:09:33.558955 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:33.558929 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9df1a4-3684-464c-97f3-21a3151a7183-ca-trust-extracted" pod "cf9df1a4-3684-464c-97f3-21a3151a7183" (UID: "cf9df1a4-3684-464c-97f3-21a3151a7183"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:09:33.653315 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:33.653283 4935 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-bound-sa-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:33.653315 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:33.653312 4935 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cf9df1a4-3684-464c-97f3-21a3151a7183-installation-pull-secrets\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:33.653315 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:33.653322 4935 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:33.653535 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:33.653331 4935 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cf9df1a4-3684-464c-97f3-21a3151a7183-trusted-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:33.653535 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:33.653340 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgxs2\" (UniqueName: \"kubernetes.io/projected/cf9df1a4-3684-464c-97f3-21a3151a7183-kube-api-access-kgxs2\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:33.653535 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:33.653349 4935 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cf9df1a4-3684-464c-97f3-21a3151a7183-registry-certificates\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:33.653535 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:33.653358 4935 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cf9df1a4-3684-464c-97f3-21a3151a7183-ca-trust-extracted\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:33.653535 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:33.653367 4935 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cf9df1a4-3684-464c-97f3-21a3151a7183-image-registry-private-configuration\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:34.121305 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:34.121261 4935 generic.go:356] "Generic (PLEG): container finished" podID="cf9df1a4-3684-464c-97f3-21a3151a7183" containerID="4c2ae0c2441064fac6a2e8a2550c8f267fac138ceb60f8ff6e473aa7337c8328" exitCode=0 Nov 02 21:09:34.121464 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:34.121315 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c987dc54-2p2h6" Nov 02 21:09:34.121464 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:34.121343 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c987dc54-2p2h6" event={"ID":"cf9df1a4-3684-464c-97f3-21a3151a7183","Type":"ContainerDied","Data":"4c2ae0c2441064fac6a2e8a2550c8f267fac138ceb60f8ff6e473aa7337c8328"} Nov 02 21:09:34.121464 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:34.121400 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c987dc54-2p2h6" event={"ID":"cf9df1a4-3684-464c-97f3-21a3151a7183","Type":"ContainerDied","Data":"62e18e11b74c4ea37ad7531f1f14f74da34f8d98ea2f483d9a7343d100f0cc07"} Nov 02 21:09:34.121464 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:34.121423 4935 scope.go:122] "RemoveContainer" containerID="4c2ae0c2441064fac6a2e8a2550c8f267fac138ceb60f8ff6e473aa7337c8328" Nov 02 21:09:34.129546 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:34.129527 4935 scope.go:122] "RemoveContainer" containerID="4c2ae0c2441064fac6a2e8a2550c8f267fac138ceb60f8ff6e473aa7337c8328" Nov 02 21:09:34.129816 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:09:34.129796 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c2ae0c2441064fac6a2e8a2550c8f267fac138ceb60f8ff6e473aa7337c8328\": container with ID starting with 4c2ae0c2441064fac6a2e8a2550c8f267fac138ceb60f8ff6e473aa7337c8328 not found: ID does not exist" containerID="4c2ae0c2441064fac6a2e8a2550c8f267fac138ceb60f8ff6e473aa7337c8328" Nov 02 21:09:34.129858 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:34.129824 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2ae0c2441064fac6a2e8a2550c8f267fac138ceb60f8ff6e473aa7337c8328"} err="failed to get container status \"4c2ae0c2441064fac6a2e8a2550c8f267fac138ceb60f8ff6e473aa7337c8328\": rpc error: code = NotFound desc = could not find container \"4c2ae0c2441064fac6a2e8a2550c8f267fac138ceb60f8ff6e473aa7337c8328\": container with ID starting with 4c2ae0c2441064fac6a2e8a2550c8f267fac138ceb60f8ff6e473aa7337c8328 not found: ID does not exist" Nov 02 21:09:34.324654 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:34.324621 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c987dc54-2p2h6"] Nov 02 21:09:34.371010 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:34.370983 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-c987dc54-2p2h6"] Nov 02 21:09:35.774686 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:35.774655 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf9df1a4-3684-464c-97f3-21a3151a7183" path="/var/lib/kubelet/pods/cf9df1a4-3684-464c-97f3-21a3151a7183/volumes" Nov 02 21:09:39.141242 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:39.141207 4935 generic.go:356] "Generic (PLEG): container finished" podID="5ca28f2b-c77d-4aa1-8501-ec380253ab88" containerID="c0adfd7d26ac62535fc6317ff0ddfbc014f0677a42b88aeba041e2fd2b04a037" exitCode=0 Nov 02 21:09:39.141696 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:39.141268 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf" event={"ID":"5ca28f2b-c77d-4aa1-8501-ec380253ab88","Type":"ContainerDied","Data":"c0adfd7d26ac62535fc6317ff0ddfbc014f0677a42b88aeba041e2fd2b04a037"} Nov 02 21:09:39.141696 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:39.141585 4935 scope.go:122] "RemoveContainer" containerID="c0adfd7d26ac62535fc6317ff0ddfbc014f0677a42b88aeba041e2fd2b04a037" Nov 02 21:09:40.146809 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:40.146774 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r7xsf" event={"ID":"5ca28f2b-c77d-4aa1-8501-ec380253ab88","Type":"ContainerStarted","Data":"c62f327f2832327aff801abab7aaf58ee9b0b64f86a954ac06d81ac5fda65183"} Nov 02 21:09:41.522976 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:41.522937 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:41.732048 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:41.732017 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:43.960108 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:43.960072 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:44.156386 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:44.156350 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:44.792457 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:44.792425 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:44.987983 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:44.987954 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:46.186068 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:46.186039 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:46.398948 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:46.398913 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:53.194948 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:53.194915 4935 generic.go:356] "Generic (PLEG): container finished" podID="8cf65e09-2079-40cc-b597-1b67ba482a04" containerID="d4b1da20e3daf85a69892deda5cd65868da955b9044216326d8e6013f8a91a44" exitCode=0 Nov 02 21:09:53.195341 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:53.194962 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-2ftg6" event={"ID":"8cf65e09-2079-40cc-b597-1b67ba482a04","Type":"ContainerDied","Data":"d4b1da20e3daf85a69892deda5cd65868da955b9044216326d8e6013f8a91a44"} Nov 02 21:09:53.195341 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:53.195277 4935 scope.go:122] "RemoveContainer" containerID="d4b1da20e3daf85a69892deda5cd65868da955b9044216326d8e6013f8a91a44" Nov 02 21:09:53.519226 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:53.519194 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:53.718389 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:53.718351 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:54.200777 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:54.200739 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-2ftg6" event={"ID":"8cf65e09-2079-40cc-b597-1b67ba482a04","Type":"ContainerStarted","Data":"58857ed9d58f747490719144b01a8c4fd8dfaffb9cd20f9a803a8db4380d5d96"} Nov 02 21:09:55.313597 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:55.313566 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:58.467816 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:09:58.467775 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:09:58.966640 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:58.966606 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:09:59.166143 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:09:59.166109 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:10:00.222965 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:10:00.222926 4935 generic.go:356] "Generic (PLEG): container finished" podID="f264bf41-80c4-4dda-a67e-6651b464655c" containerID="6dd665d446837b1ec657406834feb0025c54d6a33b5795993c88ea56dd5b1835" exitCode=0 Nov 02 21:10:00.223486 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:10:00.223002 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pfjlk" event={"ID":"f264bf41-80c4-4dda-a67e-6651b464655c","Type":"ContainerDied","Data":"6dd665d446837b1ec657406834feb0025c54d6a33b5795993c88ea56dd5b1835"} Nov 02 21:10:00.223486 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:10:00.223462 4935 scope.go:122] "RemoveContainer" containerID="6dd665d446837b1ec657406834feb0025c54d6a33b5795993c88ea56dd5b1835" Nov 02 21:10:01.231071 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:10:01.231034 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pfjlk" event={"ID":"f264bf41-80c4-4dda-a67e-6651b464655c","Type":"ContainerStarted","Data":"0c5a84f098134f6019370928d87ef39458cd34944d40575e7fe34e5a691321f7"} Nov 02 21:10:17.972706 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:10:17.972672 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:10:28.468126 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:10:28.468085 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:10:39.236683 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:10:39.236604 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:10:39.440031 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:10:39.439995 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:10:51.881522 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:10:51.881484 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:10:52.114122 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:10:52.114088 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:10:58.467049 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:10:58.467011 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:10:58.955526 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:10:58.955488 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:10:59.173511 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:10:59.173475 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:11:02.363681 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:11:02.363647 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:11:02.743326 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:11:02.743242 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:11:02.980059 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:11:02.980021 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:11:18.298779 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:11:18.298742 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:11:28.468207 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:11:28.468169 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:11:58.467511 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:11:58.467465 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:12:01.907070 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:01.907030 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:18.627475 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:18.627391 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:28.468496 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:12:28.468454 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:12:45.595849 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:45.595817 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:48.535825 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:48.535784 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:48.807856 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:48.807776 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:49.039450 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:49.039412 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:49.566570 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:49.566536 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt"] Nov 02 21:12:49.566950 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:49.566846 4935 cpu_manager.go:414] "containerMap: removing container" podUID="08b38b87-8fd7-475e-94db-8bdd1339dc4b" containerName="registry" Nov 02 21:12:49.566950 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:49.566858 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08b38b87-8fd7-475e-94db-8bdd1339dc4b" containerName="registry" Nov 02 21:12:49.566950 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:49.566883 4935 cpu_manager.go:414] "containerMap: removing container" podUID="cf9df1a4-3684-464c-97f3-21a3151a7183" containerName="registry" Nov 02 21:12:49.566950 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:49.566888 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf9df1a4-3684-464c-97f3-21a3151a7183" containerName="registry" Nov 02 21:12:49.567073 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:49.566953 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="08b38b87-8fd7-475e-94db-8bdd1339dc4b" containerName="registry" Nov 02 21:12:49.567073 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:49.566966 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf9df1a4-3684-464c-97f3-21a3151a7183" containerName="registry" Nov 02 21:12:49.570026 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:49.570009 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt" Nov 02 21:12:49.609763 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:49.609734 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aeca5d99-bcef-4f43-9991-305e841a9a9e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt\" (UID: \"aeca5d99-bcef-4f43-9991-305e841a9a9e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt" Nov 02 21:12:49.609888 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:49.609768 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aeca5d99-bcef-4f43-9991-305e841a9a9e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt\" (UID: \"aeca5d99-bcef-4f43-9991-305e841a9a9e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt" Nov 02 21:12:49.609888 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:49.609786 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6khvm\" (UniqueName: \"kubernetes.io/projected/aeca5d99-bcef-4f43-9991-305e841a9a9e-kube-api-access-6khvm\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt\" (UID: \"aeca5d99-bcef-4f43-9991-305e841a9a9e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt" Nov 02 21:12:49.665814 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:49.665787 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 21:12:49.665814 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:49.665796 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 21:12:49.665980 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:49.665795 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-cdfsj\"" Nov 02 21:12:49.666895 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:49.666877 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt"] Nov 02 21:12:49.710503 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:49.710475 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aeca5d99-bcef-4f43-9991-305e841a9a9e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt\" (UID: \"aeca5d99-bcef-4f43-9991-305e841a9a9e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt" Nov 02 21:12:49.710668 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:49.710507 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6khvm\" (UniqueName: \"kubernetes.io/projected/aeca5d99-bcef-4f43-9991-305e841a9a9e-kube-api-access-6khvm\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt\" (UID: \"aeca5d99-bcef-4f43-9991-305e841a9a9e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt" Nov 02 21:12:49.710668 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:49.710580 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aeca5d99-bcef-4f43-9991-305e841a9a9e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt\" (UID: \"aeca5d99-bcef-4f43-9991-305e841a9a9e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt" Nov 02 21:12:49.710868 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:49.710849 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aeca5d99-bcef-4f43-9991-305e841a9a9e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt\" (UID: \"aeca5d99-bcef-4f43-9991-305e841a9a9e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt" Nov 02 21:12:49.710903 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:49.710883 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aeca5d99-bcef-4f43-9991-305e841a9a9e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt\" (UID: \"aeca5d99-bcef-4f43-9991-305e841a9a9e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt" Nov 02 21:12:49.766107 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:49.766077 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6khvm\" (UniqueName: \"kubernetes.io/projected/aeca5d99-bcef-4f43-9991-305e841a9a9e-kube-api-access-6khvm\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt\" (UID: \"aeca5d99-bcef-4f43-9991-305e841a9a9e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt" Nov 02 21:12:49.885896 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:49.885823 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt" Nov 02 21:12:50.035021 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.034984 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:50.241144 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.241059 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:50.252085 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.252052 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt"] Nov 02 21:12:50.255747 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:12:50.255715 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeca5d99_bcef_4f43_9991_305e841a9a9e.slice/crio-76053cc45d54e72487497984c65d016703b2821d299fd6d9fa63ca76b92f986f WatchSource:0}: Error finding container 76053cc45d54e72487497984c65d016703b2821d299fd6d9fa63ca76b92f986f: Status 404 returned error can't find the container with id 76053cc45d54e72487497984c65d016703b2821d299fd6d9fa63ca76b92f986f Nov 02 21:12:50.637423 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.637386 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9"] Nov 02 21:12:50.641835 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.641814 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9" Nov 02 21:12:50.719032 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.718994 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9\" (UID: \"9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9" Nov 02 21:12:50.719222 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.719048 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9\" (UID: \"9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9" Nov 02 21:12:50.719222 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.719161 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cp69\" (UniqueName: \"kubernetes.io/projected/9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c-kube-api-access-6cp69\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9\" (UID: \"9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9" Nov 02 21:12:50.739540 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.739508 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-2t6z9\"" Nov 02 21:12:50.740678 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.740665 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9"] Nov 02 21:12:50.741400 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.741376 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt"] Nov 02 21:12:50.744677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.744661 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt" Nov 02 21:12:50.793910 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.793886 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-9n7nj\"" Nov 02 21:12:50.794124 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.794095 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt"] Nov 02 21:12:50.804315 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.804294 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt" event={"ID":"aeca5d99-bcef-4f43-9991-305e841a9a9e","Type":"ContainerStarted","Data":"76053cc45d54e72487497984c65d016703b2821d299fd6d9fa63ca76b92f986f"} Nov 02 21:12:50.820366 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.820343 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab8ef755-2409-4114-ac23-31937bb52b4e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt\" (UID: \"ab8ef755-2409-4114-ac23-31937bb52b4e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt" Nov 02 21:12:50.820463 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.820388 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cp69\" (UniqueName: \"kubernetes.io/projected/9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c-kube-api-access-6cp69\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9\" (UID: \"9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9" Nov 02 21:12:50.820532 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.820516 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9\" (UID: \"9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9" Nov 02 21:12:50.820585 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.820556 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab8ef755-2409-4114-ac23-31937bb52b4e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt\" (UID: \"ab8ef755-2409-4114-ac23-31937bb52b4e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt" Nov 02 21:12:50.820695 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.820659 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q2gw\" (UniqueName: \"kubernetes.io/projected/ab8ef755-2409-4114-ac23-31937bb52b4e-kube-api-access-7q2gw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt\" (UID: \"ab8ef755-2409-4114-ac23-31937bb52b4e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt" Nov 02 21:12:50.820780 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.820731 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9\" (UID: \"9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9" Nov 02 21:12:50.820907 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.820888 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9\" (UID: \"9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9" Nov 02 21:12:50.820989 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.820973 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9\" (UID: \"9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9" Nov 02 21:12:50.834738 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.834718 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh"] Nov 02 21:12:50.837960 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.837944 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh" Nov 02 21:12:50.877382 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.877353 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cp69\" (UniqueName: \"kubernetes.io/projected/9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c-kube-api-access-6cp69\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9\" (UID: \"9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9" Nov 02 21:12:50.888997 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.888945 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-pd9kp\"" Nov 02 21:12:50.891195 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.891177 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh"] Nov 02 21:12:50.921763 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.921739 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvs6g\" (UniqueName: \"kubernetes.io/projected/701c4c49-710a-4b9c-b050-d2ea23303741-kube-api-access-kvs6g\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh\" (UID: \"701c4c49-710a-4b9c-b050-d2ea23303741\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh" Nov 02 21:12:50.921873 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.921777 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/701c4c49-710a-4b9c-b050-d2ea23303741-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh\" (UID: \"701c4c49-710a-4b9c-b050-d2ea23303741\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh" Nov 02 21:12:50.921873 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.921799 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q2gw\" (UniqueName: \"kubernetes.io/projected/ab8ef755-2409-4114-ac23-31937bb52b4e-kube-api-access-7q2gw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt\" (UID: \"ab8ef755-2409-4114-ac23-31937bb52b4e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt" Nov 02 21:12:50.922098 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.921868 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab8ef755-2409-4114-ac23-31937bb52b4e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt\" (UID: \"ab8ef755-2409-4114-ac23-31937bb52b4e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt" Nov 02 21:12:50.922098 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.921938 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/701c4c49-710a-4b9c-b050-d2ea23303741-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh\" (UID: \"701c4c49-710a-4b9c-b050-d2ea23303741\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh" Nov 02 21:12:50.922098 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.921976 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab8ef755-2409-4114-ac23-31937bb52b4e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt\" (UID: \"ab8ef755-2409-4114-ac23-31937bb52b4e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt" Nov 02 21:12:50.922329 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.922311 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab8ef755-2409-4114-ac23-31937bb52b4e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt\" (UID: \"ab8ef755-2409-4114-ac23-31937bb52b4e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt" Nov 02 21:12:50.922368 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.922327 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab8ef755-2409-4114-ac23-31937bb52b4e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt\" (UID: \"ab8ef755-2409-4114-ac23-31937bb52b4e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt" Nov 02 21:12:50.937074 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.937054 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8"] Nov 02 21:12:50.940524 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.940510 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8" Nov 02 21:12:50.958024 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.958004 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9" Nov 02 21:12:50.975925 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.975900 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q2gw\" (UniqueName: \"kubernetes.io/projected/ab8ef755-2409-4114-ac23-31937bb52b4e-kube-api-access-7q2gw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt\" (UID: \"ab8ef755-2409-4114-ac23-31937bb52b4e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt" Nov 02 21:12:50.979935 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.979911 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:50.992706 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.992482 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-t5w82\"" Nov 02 21:12:50.994013 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:50.993986 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8"] Nov 02 21:12:51.022959 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.022936 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/701c4c49-710a-4b9c-b050-d2ea23303741-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh\" (UID: \"701c4c49-710a-4b9c-b050-d2ea23303741\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh" Nov 02 21:12:51.023069 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.022974 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvs6g\" (UniqueName: \"kubernetes.io/projected/701c4c49-710a-4b9c-b050-d2ea23303741-kube-api-access-kvs6g\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh\" (UID: \"701c4c49-710a-4b9c-b050-d2ea23303741\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh" Nov 02 21:12:51.023069 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.023001 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/701c4c49-710a-4b9c-b050-d2ea23303741-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh\" (UID: \"701c4c49-710a-4b9c-b050-d2ea23303741\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh" Nov 02 21:12:51.023069 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.023032 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/466e29e9-9ab4-42ec-bf83-7c781a15e084-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8\" (UID: \"466e29e9-9ab4-42ec-bf83-7c781a15e084\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8" Nov 02 21:12:51.023069 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.023053 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7m4d\" (UniqueName: \"kubernetes.io/projected/466e29e9-9ab4-42ec-bf83-7c781a15e084-kube-api-access-x7m4d\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8\" (UID: \"466e29e9-9ab4-42ec-bf83-7c781a15e084\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8" Nov 02 21:12:51.023306 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.023074 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/466e29e9-9ab4-42ec-bf83-7c781a15e084-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8\" (UID: \"466e29e9-9ab4-42ec-bf83-7c781a15e084\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8" Nov 02 21:12:51.023430 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.023408 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/701c4c49-710a-4b9c-b050-d2ea23303741-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh\" (UID: \"701c4c49-710a-4b9c-b050-d2ea23303741\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh" Nov 02 21:12:51.023491 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.023442 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/701c4c49-710a-4b9c-b050-d2ea23303741-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh\" (UID: \"701c4c49-710a-4b9c-b050-d2ea23303741\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh" Nov 02 21:12:51.059875 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.059848 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt" Nov 02 21:12:51.076461 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.076426 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvs6g\" (UniqueName: \"kubernetes.io/projected/701c4c49-710a-4b9c-b050-d2ea23303741-kube-api-access-kvs6g\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh\" (UID: \"701c4c49-710a-4b9c-b050-d2ea23303741\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh" Nov 02 21:12:51.123847 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.123814 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/466e29e9-9ab4-42ec-bf83-7c781a15e084-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8\" (UID: \"466e29e9-9ab4-42ec-bf83-7c781a15e084\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8" Nov 02 21:12:51.123974 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.123856 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7m4d\" (UniqueName: \"kubernetes.io/projected/466e29e9-9ab4-42ec-bf83-7c781a15e084-kube-api-access-x7m4d\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8\" (UID: \"466e29e9-9ab4-42ec-bf83-7c781a15e084\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8" Nov 02 21:12:51.123974 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.123878 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/466e29e9-9ab4-42ec-bf83-7c781a15e084-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8\" (UID: \"466e29e9-9ab4-42ec-bf83-7c781a15e084\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8" Nov 02 21:12:51.124280 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.124257 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/466e29e9-9ab4-42ec-bf83-7c781a15e084-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8\" (UID: \"466e29e9-9ab4-42ec-bf83-7c781a15e084\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8" Nov 02 21:12:51.124280 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.124272 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/466e29e9-9ab4-42ec-bf83-7c781a15e084-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8\" (UID: \"466e29e9-9ab4-42ec-bf83-7c781a15e084\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8" Nov 02 21:12:51.152972 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.152907 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh" Nov 02 21:12:51.177028 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.176996 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7m4d\" (UniqueName: \"kubernetes.io/projected/466e29e9-9ab4-42ec-bf83-7c781a15e084-kube-api-access-x7m4d\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8\" (UID: \"466e29e9-9ab4-42ec-bf83-7c781a15e084\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8" Nov 02 21:12:51.255647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.255613 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8" Nov 02 21:12:51.325091 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.325042 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9"] Nov 02 21:12:51.328813 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:12:51.328775 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef41d0c_1220_4c93_959b_0c4b7e1b5e9c.slice/crio-f73b7fa7d640322d4547907ef80a006b4d588b05667910ebd10c21510deef091 WatchSource:0}: Error finding container f73b7fa7d640322d4547907ef80a006b4d588b05667910ebd10c21510deef091: Status 404 returned error can't find the container with id f73b7fa7d640322d4547907ef80a006b4d588b05667910ebd10c21510deef091 Nov 02 21:12:51.425393 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.425322 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt"] Nov 02 21:12:51.429613 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:12:51.429584 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab8ef755_2409_4114_ac23_31937bb52b4e.slice/crio-da10fb407d843b1cfd506ce174fbdbeb68bab4c2d00eb84480d17b88a2cf842e WatchSource:0}: Error finding container da10fb407d843b1cfd506ce174fbdbeb68bab4c2d00eb84480d17b88a2cf842e: Status 404 returned error can't find the container with id da10fb407d843b1cfd506ce174fbdbeb68bab4c2d00eb84480d17b88a2cf842e Nov 02 21:12:51.514233 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.514201 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh"] Nov 02 21:12:51.520281 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:12:51.520251 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod701c4c49_710a_4b9c_b050_d2ea23303741.slice/crio-c1c1cb646f41477a64f1fcaa33f8bd2297793aad25149be1c4227380005af066 WatchSource:0}: Error finding container c1c1cb646f41477a64f1fcaa33f8bd2297793aad25149be1c4227380005af066: Status 404 returned error can't find the container with id c1c1cb646f41477a64f1fcaa33f8bd2297793aad25149be1c4227380005af066 Nov 02 21:12:51.618645 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.618610 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8"] Nov 02 21:12:51.621771 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:12:51.621737 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod466e29e9_9ab4_42ec_bf83_7c781a15e084.slice/crio-ebe2860e4c259431c36eaadf04912dc8267efd58f96c2e50e48fe87309f44237 WatchSource:0}: Error finding container ebe2860e4c259431c36eaadf04912dc8267efd58f96c2e50e48fe87309f44237: Status 404 returned error can't find the container with id ebe2860e4c259431c36eaadf04912dc8267efd58f96c2e50e48fe87309f44237 Nov 02 21:12:51.729974 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.729892 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b"] Nov 02 21:12:51.735383 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.735357 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b" Nov 02 21:12:51.786278 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.786246 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-6l7kd\"" Nov 02 21:12:51.787301 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.787274 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b"] Nov 02 21:12:51.810607 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.810566 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9" event={"ID":"9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c","Type":"ContainerStarted","Data":"f73b7fa7d640322d4547907ef80a006b4d588b05667910ebd10c21510deef091"} Nov 02 21:12:51.811866 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.811830 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8" event={"ID":"466e29e9-9ab4-42ec-bf83-7c781a15e084","Type":"ContainerStarted","Data":"ebe2860e4c259431c36eaadf04912dc8267efd58f96c2e50e48fe87309f44237"} Nov 02 21:12:51.813450 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.813420 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt" event={"ID":"ab8ef755-2409-4114-ac23-31937bb52b4e","Type":"ContainerStarted","Data":"da10fb407d843b1cfd506ce174fbdbeb68bab4c2d00eb84480d17b88a2cf842e"} Nov 02 21:12:51.814610 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.814588 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh" event={"ID":"701c4c49-710a-4b9c-b050-d2ea23303741","Type":"ContainerStarted","Data":"c1c1cb646f41477a64f1fcaa33f8bd2297793aad25149be1c4227380005af066"} Nov 02 21:12:51.828289 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.828262 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/829a4d78-6c46-448b-af0f-e8d8a7daefd7-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b\" (UID: \"829a4d78-6c46-448b-af0f-e8d8a7daefd7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b" Nov 02 21:12:51.828386 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.828307 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/829a4d78-6c46-448b-af0f-e8d8a7daefd7-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b\" (UID: \"829a4d78-6c46-448b-af0f-e8d8a7daefd7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b" Nov 02 21:12:51.828386 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.828340 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4n2r\" (UniqueName: \"kubernetes.io/projected/829a4d78-6c46-448b-af0f-e8d8a7daefd7-kube-api-access-r4n2r\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b\" (UID: \"829a4d78-6c46-448b-af0f-e8d8a7daefd7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b" Nov 02 21:12:51.928768 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.928730 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/829a4d78-6c46-448b-af0f-e8d8a7daefd7-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b\" (UID: \"829a4d78-6c46-448b-af0f-e8d8a7daefd7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b" Nov 02 21:12:51.928945 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.928813 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4n2r\" (UniqueName: \"kubernetes.io/projected/829a4d78-6c46-448b-af0f-e8d8a7daefd7-kube-api-access-r4n2r\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b\" (UID: \"829a4d78-6c46-448b-af0f-e8d8a7daefd7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b" Nov 02 21:12:51.928945 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.928892 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/829a4d78-6c46-448b-af0f-e8d8a7daefd7-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b\" (UID: \"829a4d78-6c46-448b-af0f-e8d8a7daefd7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b" Nov 02 21:12:51.929229 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.929204 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/829a4d78-6c46-448b-af0f-e8d8a7daefd7-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b\" (UID: \"829a4d78-6c46-448b-af0f-e8d8a7daefd7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b" Nov 02 21:12:51.929303 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.929230 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/829a4d78-6c46-448b-af0f-e8d8a7daefd7-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b\" (UID: \"829a4d78-6c46-448b-af0f-e8d8a7daefd7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b" Nov 02 21:12:51.982875 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:51.982798 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4n2r\" (UniqueName: \"kubernetes.io/projected/829a4d78-6c46-448b-af0f-e8d8a7daefd7-kube-api-access-r4n2r\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b\" (UID: \"829a4d78-6c46-448b-af0f-e8d8a7daefd7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b" Nov 02 21:12:52.054405 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:52.054373 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b" Nov 02 21:12:52.425950 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:52.425916 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b"] Nov 02 21:12:52.430948 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:12:52.430623 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod829a4d78_6c46_448b_af0f_e8d8a7daefd7.slice/crio-492636fff5ecb3d48cb1191ccc58767c4d7530a7ece54cda9d47c8bd7efb7cdd WatchSource:0}: Error finding container 492636fff5ecb3d48cb1191ccc58767c4d7530a7ece54cda9d47c8bd7efb7cdd: Status 404 returned error can't find the container with id 492636fff5ecb3d48cb1191ccc58767c4d7530a7ece54cda9d47c8bd7efb7cdd Nov 02 21:12:52.823781 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:52.823694 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b" event={"ID":"829a4d78-6c46-448b-af0f-e8d8a7daefd7","Type":"ContainerStarted","Data":"492636fff5ecb3d48cb1191ccc58767c4d7530a7ece54cda9d47c8bd7efb7cdd"} Nov 02 21:12:53.045314 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:53.045279 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w"] Nov 02 21:12:53.049815 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:53.049777 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w" Nov 02 21:12:53.137872 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:53.137791 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/112e3a47-4817-4237-b296-aa3e260584ef-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w\" (UID: \"112e3a47-4817-4237-b296-aa3e260584ef\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w" Nov 02 21:12:53.138106 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:53.137892 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcg97\" (UniqueName: \"kubernetes.io/projected/112e3a47-4817-4237-b296-aa3e260584ef-kube-api-access-dcg97\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w\" (UID: \"112e3a47-4817-4237-b296-aa3e260584ef\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w" Nov 02 21:12:53.138106 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:53.137938 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/112e3a47-4817-4237-b296-aa3e260584ef-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w\" (UID: \"112e3a47-4817-4237-b296-aa3e260584ef\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w" Nov 02 21:12:53.147065 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:53.147005 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-856q7\"" Nov 02 21:12:53.148012 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:53.147987 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w"] Nov 02 21:12:53.240919 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:53.239664 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcg97\" (UniqueName: \"kubernetes.io/projected/112e3a47-4817-4237-b296-aa3e260584ef-kube-api-access-dcg97\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w\" (UID: \"112e3a47-4817-4237-b296-aa3e260584ef\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w" Nov 02 21:12:53.240919 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:53.239714 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/112e3a47-4817-4237-b296-aa3e260584ef-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w\" (UID: \"112e3a47-4817-4237-b296-aa3e260584ef\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w" Nov 02 21:12:53.240919 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:53.239780 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/112e3a47-4817-4237-b296-aa3e260584ef-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w\" (UID: \"112e3a47-4817-4237-b296-aa3e260584ef\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w" Nov 02 21:12:53.240919 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:53.240299 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/112e3a47-4817-4237-b296-aa3e260584ef-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w\" (UID: \"112e3a47-4817-4237-b296-aa3e260584ef\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w" Nov 02 21:12:53.240919 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:53.240872 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/112e3a47-4817-4237-b296-aa3e260584ef-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w\" (UID: \"112e3a47-4817-4237-b296-aa3e260584ef\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w" Nov 02 21:12:53.301525 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:53.301485 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcg97\" (UniqueName: \"kubernetes.io/projected/112e3a47-4817-4237-b296-aa3e260584ef-kube-api-access-dcg97\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w\" (UID: \"112e3a47-4817-4237-b296-aa3e260584ef\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w" Nov 02 21:12:53.372452 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:53.372412 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w" Nov 02 21:12:53.797708 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:53.797676 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w"] Nov 02 21:12:53.816649 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:12:53.816607 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod112e3a47_4817_4237_b296_aa3e260584ef.slice/crio-44662897dfb5e38fae0efb3fff75d38be8cfb34aa5fff5adac68fd17242dc688 WatchSource:0}: Error finding container 44662897dfb5e38fae0efb3fff75d38be8cfb34aa5fff5adac68fd17242dc688: Status 404 returned error can't find the container with id 44662897dfb5e38fae0efb3fff75d38be8cfb34aa5fff5adac68fd17242dc688 Nov 02 21:12:53.837915 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:53.837876 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w" event={"ID":"112e3a47-4817-4237-b296-aa3e260584ef","Type":"ContainerStarted","Data":"44662897dfb5e38fae0efb3fff75d38be8cfb34aa5fff5adac68fd17242dc688"} Nov 02 21:12:55.821160 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:55.819039 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj"] Nov 02 21:12:55.825095 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:55.825070 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj" Nov 02 21:12:55.867293 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:55.866940 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjrzz\" (UniqueName: \"kubernetes.io/projected/aa46e508-065d-4ce1-b46a-ed1194a6e3cb-kube-api-access-pjrzz\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj\" (UID: \"aa46e508-065d-4ce1-b46a-ed1194a6e3cb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj" Nov 02 21:12:55.867293 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:55.866994 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa46e508-065d-4ce1-b46a-ed1194a6e3cb-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj\" (UID: \"aa46e508-065d-4ce1-b46a-ed1194a6e3cb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj" Nov 02 21:12:55.867293 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:55.867044 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa46e508-065d-4ce1-b46a-ed1194a6e3cb-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj\" (UID: \"aa46e508-065d-4ce1-b46a-ed1194a6e3cb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj" Nov 02 21:12:55.875538 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:55.875503 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj"] Nov 02 21:12:55.969342 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:55.967974 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjrzz\" (UniqueName: \"kubernetes.io/projected/aa46e508-065d-4ce1-b46a-ed1194a6e3cb-kube-api-access-pjrzz\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj\" (UID: \"aa46e508-065d-4ce1-b46a-ed1194a6e3cb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj" Nov 02 21:12:55.969342 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:55.968028 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa46e508-065d-4ce1-b46a-ed1194a6e3cb-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj\" (UID: \"aa46e508-065d-4ce1-b46a-ed1194a6e3cb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj" Nov 02 21:12:55.969342 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:55.968072 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa46e508-065d-4ce1-b46a-ed1194a6e3cb-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj\" (UID: \"aa46e508-065d-4ce1-b46a-ed1194a6e3cb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj" Nov 02 21:12:55.969342 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:55.968629 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa46e508-065d-4ce1-b46a-ed1194a6e3cb-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj\" (UID: \"aa46e508-065d-4ce1-b46a-ed1194a6e3cb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj" Nov 02 21:12:55.969342 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:55.969220 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa46e508-065d-4ce1-b46a-ed1194a6e3cb-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj\" (UID: \"aa46e508-065d-4ce1-b46a-ed1194a6e3cb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj" Nov 02 21:12:56.023010 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:56.022941 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjrzz\" (UniqueName: \"kubernetes.io/projected/aa46e508-065d-4ce1-b46a-ed1194a6e3cb-kube-api-access-pjrzz\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj\" (UID: \"aa46e508-065d-4ce1-b46a-ed1194a6e3cb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj" Nov 02 21:12:56.147489 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:56.147404 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj" Nov 02 21:12:56.423764 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:56.423675 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:56.590178 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:56.590121 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj"] Nov 02 21:12:56.592518 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:12:56.592487 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa46e508_065d_4ce1_b46a_ed1194a6e3cb.slice/crio-3420e10f50f5b3335efd055c5fd01b51cf2485acb8757683e9e6c74710e414b9 WatchSource:0}: Error finding container 3420e10f50f5b3335efd055c5fd01b51cf2485acb8757683e9e6c74710e414b9: Status 404 returned error can't find the container with id 3420e10f50f5b3335efd055c5fd01b51cf2485acb8757683e9e6c74710e414b9 Nov 02 21:12:56.645844 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:56.645807 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:56.863915 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:56.863876 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj" event={"ID":"aa46e508-065d-4ce1-b46a-ed1194a6e3cb","Type":"ContainerStarted","Data":"3420e10f50f5b3335efd055c5fd01b51cf2485acb8757683e9e6c74710e414b9"} Nov 02 21:12:57.259028 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.258944 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4"] Nov 02 21:12:57.263809 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.263773 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.282035 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.281547 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9f99d96-f28e-4152-8285-c6aefa58a412-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4r6s4\" (UID: \"d9f99d96-f28e-4152-8285-c6aefa58a412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.282035 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.281609 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f99d96-f28e-4152-8285-c6aefa58a412-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4r6s4\" (UID: \"d9f99d96-f28e-4152-8285-c6aefa58a412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.282035 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.281638 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f99d96-f28e-4152-8285-c6aefa58a412-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4r6s4\" (UID: \"d9f99d96-f28e-4152-8285-c6aefa58a412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.282035 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.281665 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9f99d96-f28e-4152-8285-c6aefa58a412-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4r6s4\" (UID: \"d9f99d96-f28e-4152-8285-c6aefa58a412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.282035 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.281689 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9f99d96-f28e-4152-8285-c6aefa58a412-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4r6s4\" (UID: \"d9f99d96-f28e-4152-8285-c6aefa58a412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.282035 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.281737 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9f99d96-f28e-4152-8285-c6aefa58a412-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4r6s4\" (UID: \"d9f99d96-f28e-4152-8285-c6aefa58a412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.282035 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.281762 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8ldb\" (UniqueName: \"kubernetes.io/projected/d9f99d96-f28e-4152-8285-c6aefa58a412-kube-api-access-f8ldb\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4r6s4\" (UID: \"d9f99d96-f28e-4152-8285-c6aefa58a412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.282035 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.281799 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9f99d96-f28e-4152-8285-c6aefa58a412-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4r6s4\" (UID: \"d9f99d96-f28e-4152-8285-c6aefa58a412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.282035 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.281845 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9f99d96-f28e-4152-8285-c6aefa58a412-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4r6s4\" (UID: \"d9f99d96-f28e-4152-8285-c6aefa58a412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.317249 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.317091 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 21:12:57.331324 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.331256 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-h4lw9\"" Nov 02 21:12:57.370057 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.370025 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4"] Nov 02 21:12:57.384712 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.382718 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9f99d96-f28e-4152-8285-c6aefa58a412-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4r6s4\" (UID: \"d9f99d96-f28e-4152-8285-c6aefa58a412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.384712 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.382782 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9f99d96-f28e-4152-8285-c6aefa58a412-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4r6s4\" (UID: \"d9f99d96-f28e-4152-8285-c6aefa58a412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.384712 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.382825 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9f99d96-f28e-4152-8285-c6aefa58a412-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4r6s4\" (UID: \"d9f99d96-f28e-4152-8285-c6aefa58a412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.384712 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.382873 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f99d96-f28e-4152-8285-c6aefa58a412-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4r6s4\" (UID: \"d9f99d96-f28e-4152-8285-c6aefa58a412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.384712 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.382898 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f99d96-f28e-4152-8285-c6aefa58a412-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4r6s4\" (UID: \"d9f99d96-f28e-4152-8285-c6aefa58a412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.384712 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.382922 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9f99d96-f28e-4152-8285-c6aefa58a412-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4r6s4\" (UID: \"d9f99d96-f28e-4152-8285-c6aefa58a412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.384712 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.382945 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9f99d96-f28e-4152-8285-c6aefa58a412-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4r6s4\" (UID: \"d9f99d96-f28e-4152-8285-c6aefa58a412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.384712 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.382993 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9f99d96-f28e-4152-8285-c6aefa58a412-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4r6s4\" (UID: \"d9f99d96-f28e-4152-8285-c6aefa58a412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.384712 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.383024 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8ldb\" (UniqueName: \"kubernetes.io/projected/d9f99d96-f28e-4152-8285-c6aefa58a412-kube-api-access-f8ldb\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4r6s4\" (UID: \"d9f99d96-f28e-4152-8285-c6aefa58a412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.384712 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.383689 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9f99d96-f28e-4152-8285-c6aefa58a412-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4r6s4\" (UID: \"d9f99d96-f28e-4152-8285-c6aefa58a412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.384712 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.384181 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f99d96-f28e-4152-8285-c6aefa58a412-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4r6s4\" (UID: \"d9f99d96-f28e-4152-8285-c6aefa58a412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.384712 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.384434 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9f99d96-f28e-4152-8285-c6aefa58a412-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4r6s4\" (UID: \"d9f99d96-f28e-4152-8285-c6aefa58a412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.384712 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.384674 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f99d96-f28e-4152-8285-c6aefa58a412-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4r6s4\" (UID: \"d9f99d96-f28e-4152-8285-c6aefa58a412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.385537 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.385331 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9f99d96-f28e-4152-8285-c6aefa58a412-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4r6s4\" (UID: \"d9f99d96-f28e-4152-8285-c6aefa58a412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.400126 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.400098 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9f99d96-f28e-4152-8285-c6aefa58a412-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4r6s4\" (UID: \"d9f99d96-f28e-4152-8285-c6aefa58a412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.406742 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.406678 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9f99d96-f28e-4152-8285-c6aefa58a412-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4r6s4\" (UID: \"d9f99d96-f28e-4152-8285-c6aefa58a412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.426092 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.426028 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:57.445076 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.444551 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9f99d96-f28e-4152-8285-c6aefa58a412-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4r6s4\" (UID: \"d9f99d96-f28e-4152-8285-c6aefa58a412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.445076 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.445036 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8ldb\" (UniqueName: \"kubernetes.io/projected/d9f99d96-f28e-4152-8285-c6aefa58a412-kube-api-access-f8ldb\") pod \"maas-default-gateway-openshift-default-fcb75c45f-4r6s4\" (UID: \"d9f99d96-f28e-4152-8285-c6aefa58a412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.549290 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.549252 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr"] Nov 02 21:12:57.554571 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.554545 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:57.582173 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.581880 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:12:57.585773 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.584592 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8acbb0a-9fb9-494f-97d3-05385082fae6-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr\" (UID: \"c8acbb0a-9fb9-494f-97d3-05385082fae6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:57.585773 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.584639 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8acbb0a-9fb9-494f-97d3-05385082fae6-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr\" (UID: \"c8acbb0a-9fb9-494f-97d3-05385082fae6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:57.585773 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.584697 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8acbb0a-9fb9-494f-97d3-05385082fae6-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr\" (UID: \"c8acbb0a-9fb9-494f-97d3-05385082fae6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:57.585773 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.584794 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8acbb0a-9fb9-494f-97d3-05385082fae6-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr\" (UID: \"c8acbb0a-9fb9-494f-97d3-05385082fae6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:57.585773 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.584821 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8acbb0a-9fb9-494f-97d3-05385082fae6-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr\" (UID: \"c8acbb0a-9fb9-494f-97d3-05385082fae6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:57.585773 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.584851 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b87qp\" (UniqueName: \"kubernetes.io/projected/c8acbb0a-9fb9-494f-97d3-05385082fae6-kube-api-access-b87qp\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr\" (UID: \"c8acbb0a-9fb9-494f-97d3-05385082fae6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:57.585773 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.584883 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8acbb0a-9fb9-494f-97d3-05385082fae6-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr\" (UID: \"c8acbb0a-9fb9-494f-97d3-05385082fae6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:57.585773 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.584929 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8acbb0a-9fb9-494f-97d3-05385082fae6-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr\" (UID: \"c8acbb0a-9fb9-494f-97d3-05385082fae6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:57.585773 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.584960 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8acbb0a-9fb9-494f-97d3-05385082fae6-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr\" (UID: \"c8acbb0a-9fb9-494f-97d3-05385082fae6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:57.609392 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.605356 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-q2n87\"" Nov 02 21:12:57.611199 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.610519 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr"] Nov 02 21:12:57.646266 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.646183 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:57.693425 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.689329 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8acbb0a-9fb9-494f-97d3-05385082fae6-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr\" (UID: \"c8acbb0a-9fb9-494f-97d3-05385082fae6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:57.693425 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.689382 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8acbb0a-9fb9-494f-97d3-05385082fae6-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr\" (UID: \"c8acbb0a-9fb9-494f-97d3-05385082fae6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:57.693425 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.689425 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8acbb0a-9fb9-494f-97d3-05385082fae6-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr\" (UID: \"c8acbb0a-9fb9-494f-97d3-05385082fae6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:57.693425 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.689451 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8acbb0a-9fb9-494f-97d3-05385082fae6-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr\" (UID: \"c8acbb0a-9fb9-494f-97d3-05385082fae6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:57.693425 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.689499 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8acbb0a-9fb9-494f-97d3-05385082fae6-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr\" (UID: \"c8acbb0a-9fb9-494f-97d3-05385082fae6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:57.693425 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.689585 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8acbb0a-9fb9-494f-97d3-05385082fae6-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr\" (UID: \"c8acbb0a-9fb9-494f-97d3-05385082fae6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:57.693425 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.689610 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8acbb0a-9fb9-494f-97d3-05385082fae6-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr\" (UID: \"c8acbb0a-9fb9-494f-97d3-05385082fae6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:57.693425 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.689639 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b87qp\" (UniqueName: \"kubernetes.io/projected/c8acbb0a-9fb9-494f-97d3-05385082fae6-kube-api-access-b87qp\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr\" (UID: \"c8acbb0a-9fb9-494f-97d3-05385082fae6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:57.693425 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.689671 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8acbb0a-9fb9-494f-97d3-05385082fae6-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr\" (UID: \"c8acbb0a-9fb9-494f-97d3-05385082fae6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:57.693425 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.690098 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8acbb0a-9fb9-494f-97d3-05385082fae6-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr\" (UID: \"c8acbb0a-9fb9-494f-97d3-05385082fae6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:57.693425 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.691751 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8acbb0a-9fb9-494f-97d3-05385082fae6-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr\" (UID: \"c8acbb0a-9fb9-494f-97d3-05385082fae6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:57.693425 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.693419 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8acbb0a-9fb9-494f-97d3-05385082fae6-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr\" (UID: \"c8acbb0a-9fb9-494f-97d3-05385082fae6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:57.694901 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.694619 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8acbb0a-9fb9-494f-97d3-05385082fae6-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr\" (UID: \"c8acbb0a-9fb9-494f-97d3-05385082fae6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:57.694901 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.694834 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8acbb0a-9fb9-494f-97d3-05385082fae6-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr\" (UID: \"c8acbb0a-9fb9-494f-97d3-05385082fae6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:57.696032 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.696010 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8acbb0a-9fb9-494f-97d3-05385082fae6-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr\" (UID: \"c8acbb0a-9fb9-494f-97d3-05385082fae6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:57.697273 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.697227 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8acbb0a-9fb9-494f-97d3-05385082fae6-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr\" (UID: \"c8acbb0a-9fb9-494f-97d3-05385082fae6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:57.785797 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.785757 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8acbb0a-9fb9-494f-97d3-05385082fae6-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr\" (UID: \"c8acbb0a-9fb9-494f-97d3-05385082fae6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:57.786125 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.786095 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b87qp\" (UniqueName: \"kubernetes.io/projected/c8acbb0a-9fb9-494f-97d3-05385082fae6-kube-api-access-b87qp\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr\" (UID: \"c8acbb0a-9fb9-494f-97d3-05385082fae6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:57.869312 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.869220 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:12:57.871648 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:57.871617 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:12:58.045702 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:58.045629 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4"] Nov 02 21:12:58.303029 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:58.302991 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr"] Nov 02 21:12:58.307489 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:12:58.307439 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8acbb0a_9fb9_494f_97d3_05385082fae6.slice/crio-e5d4cafb203ba709669315ab1d454ead97f0d0416c40dbd5784cd188fae19b77 WatchSource:0}: Error finding container e5d4cafb203ba709669315ab1d454ead97f0d0416c40dbd5784cd188fae19b77: Status 404 returned error can't find the container with id e5d4cafb203ba709669315ab1d454ead97f0d0416c40dbd5784cd188fae19b77 Nov 02 21:12:58.533466 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:12:58.533427 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:12:58.884287 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:58.884245 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" event={"ID":"c8acbb0a-9fb9-494f-97d3-05385082fae6","Type":"ContainerStarted","Data":"e5d4cafb203ba709669315ab1d454ead97f0d0416c40dbd5784cd188fae19b77"} Nov 02 21:12:58.886836 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:12:58.886804 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" event={"ID":"d9f99d96-f28e-4152-8285-c6aefa58a412","Type":"ContainerStarted","Data":"691ead6822f773e923d688251bd85560608623025685444c8a35b98ae20e6e71"} Nov 02 21:13:01.881432 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:01.881366 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:05.954422 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:05.954362 4935 generic.go:356] "Generic (PLEG): container finished" podID="aeca5d99-bcef-4f43-9991-305e841a9a9e" containerID="95c97f860ef3eb5dd1ee87486476be2fa2c59af310c8d13a210b6285effa7c4c" exitCode=0 Nov 02 21:13:05.954898 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:05.954563 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt" event={"ID":"aeca5d99-bcef-4f43-9991-305e841a9a9e","Type":"ContainerDied","Data":"95c97f860ef3eb5dd1ee87486476be2fa2c59af310c8d13a210b6285effa7c4c"} Nov 02 21:13:05.958397 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:05.958371 4935 generic.go:356] "Generic (PLEG): container finished" podID="112e3a47-4817-4237-b296-aa3e260584ef" containerID="ef7e4f6264a8e1d453126bd6ca92ca6025c3ce1544bb8fb87a12fcac3d89d100" exitCode=0 Nov 02 21:13:05.958502 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:05.958407 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w" event={"ID":"112e3a47-4817-4237-b296-aa3e260584ef","Type":"ContainerDied","Data":"ef7e4f6264a8e1d453126bd6ca92ca6025c3ce1544bb8fb87a12fcac3d89d100"} Nov 02 21:13:08.518178 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:08.518125 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:13:08.518648 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:08.518230 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:13:08.518648 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:08.518272 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:13:08.596790 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:08.596754 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:13:08.596912 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:08.596842 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:13:08.596912 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:08.596873 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:13:08.675702 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:13:08.675667 4935 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod829a4d78_6c46_448b_af0f_e8d8a7daefd7.slice/crio-a40e6504404f8c3dcac3ecf9a6e0ea8658618b630afaf218ec823a9b7143598d.scope\": RecentStats: unable to find data in memory cache]" Nov 02 21:13:08.992609 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:08.992567 4935 generic.go:356] "Generic (PLEG): container finished" podID="aa46e508-065d-4ce1-b46a-ed1194a6e3cb" containerID="9c1ac4c9b7fdcdb0e0ec04323a271b1eadb008d26004536ac48ddaeaf7f78dc0" exitCode=0 Nov 02 21:13:08.992800 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:08.992642 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj" event={"ID":"aa46e508-065d-4ce1-b46a-ed1194a6e3cb","Type":"ContainerDied","Data":"9c1ac4c9b7fdcdb0e0ec04323a271b1eadb008d26004536ac48ddaeaf7f78dc0"} Nov 02 21:13:08.994773 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:08.994746 4935 generic.go:356] "Generic (PLEG): container finished" podID="701c4c49-710a-4b9c-b050-d2ea23303741" containerID="e45e75f97bb5efd66c47756efef87728edca85097fdcceadb95990cb27a67276" exitCode=0 Nov 02 21:13:08.994891 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:08.994847 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh" event={"ID":"701c4c49-710a-4b9c-b050-d2ea23303741","Type":"ContainerDied","Data":"e45e75f97bb5efd66c47756efef87728edca85097fdcceadb95990cb27a67276"} Nov 02 21:13:08.997300 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:08.997276 4935 generic.go:356] "Generic (PLEG): container finished" podID="829a4d78-6c46-448b-af0f-e8d8a7daefd7" containerID="a40e6504404f8c3dcac3ecf9a6e0ea8658618b630afaf218ec823a9b7143598d" exitCode=0 Nov 02 21:13:08.997524 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:08.997502 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b" event={"ID":"829a4d78-6c46-448b-af0f-e8d8a7daefd7","Type":"ContainerDied","Data":"a40e6504404f8c3dcac3ecf9a6e0ea8658618b630afaf218ec823a9b7143598d"} Nov 02 21:13:08.999566 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:08.999540 4935 generic.go:356] "Generic (PLEG): container finished" podID="9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c" containerID="adee9bcb58e90ee8125108f5b0fe7acdbbd7e23b54773c9b95cd2c74ebac67c6" exitCode=0 Nov 02 21:13:08.999645 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:08.999612 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9" event={"ID":"9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c","Type":"ContainerDied","Data":"adee9bcb58e90ee8125108f5b0fe7acdbbd7e23b54773c9b95cd2c74ebac67c6"} Nov 02 21:13:09.003883 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:09.003839 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" event={"ID":"c8acbb0a-9fb9-494f-97d3-05385082fae6","Type":"ContainerStarted","Data":"325ca21993ec09dbde90643b8cc6901c53e77462ba7537ff8912c3d61755a256"} Nov 02 21:13:09.006519 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:09.006495 4935 generic.go:356] "Generic (PLEG): container finished" podID="466e29e9-9ab4-42ec-bf83-7c781a15e084" containerID="be69936bcfc5f3d985f34afb2a0bf0a59a5a364cfb854b3923a5976f98b6590c" exitCode=0 Nov 02 21:13:09.006625 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:09.006568 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8" event={"ID":"466e29e9-9ab4-42ec-bf83-7c781a15e084","Type":"ContainerDied","Data":"be69936bcfc5f3d985f34afb2a0bf0a59a5a364cfb854b3923a5976f98b6590c"} Nov 02 21:13:09.009733 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:09.009706 4935 generic.go:356] "Generic (PLEG): container finished" podID="ab8ef755-2409-4114-ac23-31937bb52b4e" containerID="f8ce1f82b948d66072b2860ab9a28d87a077d9f3485e2f8ff46d44d6dd90ea9b" exitCode=0 Nov 02 21:13:09.009838 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:09.009795 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt" event={"ID":"ab8ef755-2409-4114-ac23-31937bb52b4e","Type":"ContainerDied","Data":"f8ce1f82b948d66072b2860ab9a28d87a077d9f3485e2f8ff46d44d6dd90ea9b"} Nov 02 21:13:09.013589 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:09.013564 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" event={"ID":"d9f99d96-f28e-4152-8285-c6aefa58a412","Type":"ContainerStarted","Data":"8ec2bc86e31614b7af498248e22d49ba25a70a0e41d1293906ecefddc80fa3b9"} Nov 02 21:13:09.450106 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:09.450052 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" podStartSLOduration=1.98894167 podStartE2EDuration="12.450032778s" totalImagesPullingTime="10.461091108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:09.448518695 +0000 UTC m=+281.888093417" watchObservedRunningTime="2026-11-02 21:13:09.450032778 +0000 UTC m=+281.889607500" Nov 02 21:13:09.582417 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:09.582386 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:13:09.588150 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:09.588097 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:13:09.667211 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:09.667116 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" podStartSLOduration=2.385823479 podStartE2EDuration="12.667095928s" totalImagesPullingTime="10.281272449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:09.665463219 +0000 UTC m=+282.105037942" watchObservedRunningTime="2026-11-02 21:13:09.667095928 +0000 UTC m=+282.106670650" Nov 02 21:13:09.872504 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:09.872469 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:13:09.877936 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:09.877874 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:13:10.018901 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:10.018867 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:13:10.019082 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:10.018931 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:13:10.020036 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:10.020008 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:13:10.020182 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:10.020097 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr" Nov 02 21:13:11.026200 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:11.026164 4935 generic.go:356] "Generic (PLEG): container finished" podID="466e29e9-9ab4-42ec-bf83-7c781a15e084" containerID="2345f868c0c2ea3e152ec73177baeff2aa3ae837b8b45b9568c33ccb6c70f34d" exitCode=0 Nov 02 21:13:11.026622 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:11.026282 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8" event={"ID":"466e29e9-9ab4-42ec-bf83-7c781a15e084","Type":"ContainerDied","Data":"2345f868c0c2ea3e152ec73177baeff2aa3ae837b8b45b9568c33ccb6c70f34d"} Nov 02 21:13:11.028967 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:11.028939 4935 generic.go:356] "Generic (PLEG): container finished" podID="ab8ef755-2409-4114-ac23-31937bb52b4e" containerID="ec29364ff300f11c7c77e2e417b5c923cebbe41cb130623b90bf79220539b775" exitCode=0 Nov 02 21:13:11.029254 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:11.029186 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt" event={"ID":"ab8ef755-2409-4114-ac23-31937bb52b4e","Type":"ContainerDied","Data":"ec29364ff300f11c7c77e2e417b5c923cebbe41cb130623b90bf79220539b775"} Nov 02 21:13:11.033096 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:11.032997 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh" event={"ID":"701c4c49-710a-4b9c-b050-d2ea23303741","Type":"ContainerStarted","Data":"7b1d1e903593941f26ba839cb440495d003004b4590b9752eaa77ec5738623e4"} Nov 02 21:13:11.035391 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:11.035365 4935 generic.go:356] "Generic (PLEG): container finished" podID="112e3a47-4817-4237-b296-aa3e260584ef" containerID="e6f1f21bca23d11995b00e37c76bef240794047e5502f7b5ea1206a7167768f8" exitCode=0 Nov 02 21:13:11.035495 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:11.035450 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w" event={"ID":"112e3a47-4817-4237-b296-aa3e260584ef","Type":"ContainerDied","Data":"e6f1f21bca23d11995b00e37c76bef240794047e5502f7b5ea1206a7167768f8"} Nov 02 21:13:11.038091 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:11.038067 4935 generic.go:356] "Generic (PLEG): container finished" podID="aeca5d99-bcef-4f43-9991-305e841a9a9e" containerID="6d74c2e66e7f6d298011b060ef2440e800af15537f4c4f1ac55724784d360556" exitCode=0 Nov 02 21:13:11.038224 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:11.038164 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt" event={"ID":"aeca5d99-bcef-4f43-9991-305e841a9a9e","Type":"ContainerDied","Data":"6d74c2e66e7f6d298011b060ef2440e800af15537f4c4f1ac55724784d360556"} Nov 02 21:13:11.040427 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:11.040403 4935 generic.go:356] "Generic (PLEG): container finished" podID="829a4d78-6c46-448b-af0f-e8d8a7daefd7" containerID="160642234d9d9efd54dd05873a75046347a12c98210e02340238965008462fdc" exitCode=0 Nov 02 21:13:11.040532 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:11.040505 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b" event={"ID":"829a4d78-6c46-448b-af0f-e8d8a7daefd7","Type":"ContainerDied","Data":"160642234d9d9efd54dd05873a75046347a12c98210e02340238965008462fdc"} Nov 02 21:13:11.043437 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:11.043330 4935 generic.go:356] "Generic (PLEG): container finished" podID="9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c" containerID="1fe6c98dcbe907d277adec3b984c7c7fc4940391a295eea5809544f93e64e967" exitCode=0 Nov 02 21:13:11.044307 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:11.044284 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9" event={"ID":"9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c","Type":"ContainerDied","Data":"1fe6c98dcbe907d277adec3b984c7c7fc4940391a295eea5809544f93e64e967"} Nov 02 21:13:12.050555 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:12.050517 4935 generic.go:356] "Generic (PLEG): container finished" podID="aa46e508-065d-4ce1-b46a-ed1194a6e3cb" containerID="1bb471e8142930e36afe903345a9184bd4a1aeb7dfd6a91f8f03306ef0484e52" exitCode=0 Nov 02 21:13:12.051085 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:12.050608 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj" event={"ID":"aa46e508-065d-4ce1-b46a-ed1194a6e3cb","Type":"ContainerDied","Data":"1bb471e8142930e36afe903345a9184bd4a1aeb7dfd6a91f8f03306ef0484e52"} Nov 02 21:13:12.053531 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:12.053341 4935 generic.go:356] "Generic (PLEG): container finished" podID="701c4c49-710a-4b9c-b050-d2ea23303741" containerID="7b1d1e903593941f26ba839cb440495d003004b4590b9752eaa77ec5738623e4" exitCode=0 Nov 02 21:13:12.053531 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:12.053406 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh" event={"ID":"701c4c49-710a-4b9c-b050-d2ea23303741","Type":"ContainerDied","Data":"7b1d1e903593941f26ba839cb440495d003004b4590b9752eaa77ec5738623e4"} Nov 02 21:13:18.976834 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:18.976747 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:27.819806 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:27.819760 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 21:13:27.822142 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:27.821971 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 21:13:27.840338 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:27.840309 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 21:13:27.841683 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:27.841661 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 21:13:28.483399 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:13:28.483359 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:13:29.111513 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:29.111460 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:29.245305 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:29.244949 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt" event={"ID":"aeca5d99-bcef-4f43-9991-305e841a9a9e","Type":"ContainerStarted","Data":"fc54e01a628dac4f2eed431720227c94fc10c27493cfefe0a1f19207e269a3aa"} Nov 02 21:13:29.247467 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:29.247439 4935 generic.go:356] "Generic (PLEG): container finished" podID="829a4d78-6c46-448b-af0f-e8d8a7daefd7" containerID="03a66a37d0931237cef1e5e44825c986606a0981b6af00ae439d4612cfd35786" exitCode=0 Nov 02 21:13:29.247589 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:29.247521 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b" event={"ID":"829a4d78-6c46-448b-af0f-e8d8a7daefd7","Type":"ContainerDied","Data":"03a66a37d0931237cef1e5e44825c986606a0981b6af00ae439d4612cfd35786"} Nov 02 21:13:29.250062 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:29.250035 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9" event={"ID":"9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c","Type":"ContainerStarted","Data":"7661f6efc4be0034e451fe751668c1f24b9acff9ea9ba86079b437fdb7b39f5f"} Nov 02 21:13:29.252673 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:29.252648 4935 generic.go:356] "Generic (PLEG): container finished" podID="466e29e9-9ab4-42ec-bf83-7c781a15e084" containerID="cf1110c4ff39f8154100862a509b4f9b3e4f3a9d4d0d781cfac2906c566e5e60" exitCode=0 Nov 02 21:13:29.252836 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:29.252814 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8" event={"ID":"466e29e9-9ab4-42ec-bf83-7c781a15e084","Type":"ContainerDied","Data":"cf1110c4ff39f8154100862a509b4f9b3e4f3a9d4d0d781cfac2906c566e5e60"} Nov 02 21:13:29.256917 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:29.256896 4935 generic.go:356] "Generic (PLEG): container finished" podID="ab8ef755-2409-4114-ac23-31937bb52b4e" containerID="d8c633d2a1754e163d96088247e52aa66c4ebf4d1ab2a11c2cdac8babba19611" exitCode=0 Nov 02 21:13:29.257016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:29.256977 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt" event={"ID":"ab8ef755-2409-4114-ac23-31937bb52b4e","Type":"ContainerDied","Data":"d8c633d2a1754e163d96088247e52aa66c4ebf4d1ab2a11c2cdac8babba19611"} Nov 02 21:13:29.259279 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:29.259256 4935 generic.go:356] "Generic (PLEG): container finished" podID="112e3a47-4817-4237-b296-aa3e260584ef" containerID="a44af205423c3985ec28e32d0b3609e40ede84ca180202d922bb7d29684b9b27" exitCode=0 Nov 02 21:13:29.259362 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:29.259296 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w" event={"ID":"112e3a47-4817-4237-b296-aa3e260584ef","Type":"ContainerDied","Data":"a44af205423c3985ec28e32d0b3609e40ede84ca180202d922bb7d29684b9b27"} Nov 02 21:13:29.510620 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:29.510553 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt" podStartSLOduration=2.76473246 podStartE2EDuration="40.51053834s" totalImagesPullingTime="37.74580588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:12:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:29.401585108 +0000 UTC m=+301.841159831" watchObservedRunningTime="2026-11-02 21:13:29.51053834 +0000 UTC m=+301.950113039" Nov 02 21:13:29.870996 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:29.870884 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9" podStartSLOduration=3.605514471 podStartE2EDuration="39.870865002s" totalImagesPullingTime="36.265350531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:12:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:29.869908793 +0000 UTC m=+302.309483495" watchObservedRunningTime="2026-11-02 21:13:29.870865002 +0000 UTC m=+302.310439736" Nov 02 21:13:30.069597 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.069557 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:30.266045 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.266016 4935 generic.go:356] "Generic (PLEG): container finished" podID="aa46e508-065d-4ce1-b46a-ed1194a6e3cb" containerID="e6b5503160c4a80c4df141409e8d24156c585023043a8150e151330dd655489c" exitCode=0 Nov 02 21:13:30.266492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.266104 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj" event={"ID":"aa46e508-065d-4ce1-b46a-ed1194a6e3cb","Type":"ContainerDied","Data":"e6b5503160c4a80c4df141409e8d24156c585023043a8150e151330dd655489c"} Nov 02 21:13:30.268256 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.268232 4935 generic.go:356] "Generic (PLEG): container finished" podID="701c4c49-710a-4b9c-b050-d2ea23303741" containerID="940ddcf0250e779172b97b9221ac48ac74041ee3d86dfde75bcdc16752c49eb3" exitCode=0 Nov 02 21:13:30.268376 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.268307 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh" event={"ID":"701c4c49-710a-4b9c-b050-d2ea23303741","Type":"ContainerDied","Data":"940ddcf0250e779172b97b9221ac48ac74041ee3d86dfde75bcdc16752c49eb3"} Nov 02 21:13:30.270268 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.270249 4935 generic.go:356] "Generic (PLEG): container finished" podID="aeca5d99-bcef-4f43-9991-305e841a9a9e" containerID="fc54e01a628dac4f2eed431720227c94fc10c27493cfefe0a1f19207e269a3aa" exitCode=0 Nov 02 21:13:30.270352 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.270269 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt" event={"ID":"aeca5d99-bcef-4f43-9991-305e841a9a9e","Type":"ContainerDied","Data":"fc54e01a628dac4f2eed431720227c94fc10c27493cfefe0a1f19207e269a3aa"} Nov 02 21:13:30.272341 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.272320 4935 generic.go:356] "Generic (PLEG): container finished" podID="9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c" containerID="7661f6efc4be0034e451fe751668c1f24b9acff9ea9ba86079b437fdb7b39f5f" exitCode=0 Nov 02 21:13:30.272419 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.272386 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9" event={"ID":"9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c","Type":"ContainerDied","Data":"7661f6efc4be0034e451fe751668c1f24b9acff9ea9ba86079b437fdb7b39f5f"} Nov 02 21:13:30.403074 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.403045 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt" Nov 02 21:13:30.485056 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.485030 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8" Nov 02 21:13:30.488523 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.488499 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b" Nov 02 21:13:30.492195 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.492176 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w" Nov 02 21:13:30.535220 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.535150 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab8ef755-2409-4114-ac23-31937bb52b4e-util\" (UniqueName: \"kubernetes.io/empty-dir/ab8ef755-2409-4114-ac23-31937bb52b4e-util\") pod \"ab8ef755-2409-4114-ac23-31937bb52b4e\" (UID: \"ab8ef755-2409-4114-ac23-31937bb52b4e\") " Nov 02 21:13:30.535220 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.535208 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab8ef755-2409-4114-ac23-31937bb52b4e-kube-api-access-7q2gw\" (UniqueName: \"kubernetes.io/projected/ab8ef755-2409-4114-ac23-31937bb52b4e-kube-api-access-7q2gw\") pod \"ab8ef755-2409-4114-ac23-31937bb52b4e\" (UID: \"ab8ef755-2409-4114-ac23-31937bb52b4e\") " Nov 02 21:13:30.535388 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.535234 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab8ef755-2409-4114-ac23-31937bb52b4e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab8ef755-2409-4114-ac23-31937bb52b4e-bundle\") pod \"ab8ef755-2409-4114-ac23-31937bb52b4e\" (UID: \"ab8ef755-2409-4114-ac23-31937bb52b4e\") " Nov 02 21:13:30.535699 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.535677 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab8ef755-2409-4114-ac23-31937bb52b4e-bundle" pod "ab8ef755-2409-4114-ac23-31937bb52b4e" (UID: "ab8ef755-2409-4114-ac23-31937bb52b4e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:30.537385 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.537362 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab8ef755-2409-4114-ac23-31937bb52b4e-kube-api-access-7q2gw" pod "ab8ef755-2409-4114-ac23-31937bb52b4e" (UID: "ab8ef755-2409-4114-ac23-31937bb52b4e"). InnerVolumeSpecName "kube-api-access-7q2gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:13:30.541375 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.541353 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab8ef755-2409-4114-ac23-31937bb52b4e-util" pod "ab8ef755-2409-4114-ac23-31937bb52b4e" (UID: "ab8ef755-2409-4114-ac23-31937bb52b4e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:30.635962 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.635930 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/829a4d78-6c46-448b-af0f-e8d8a7daefd7-kube-api-access-r4n2r\" (UniqueName: \"kubernetes.io/projected/829a4d78-6c46-448b-af0f-e8d8a7daefd7-kube-api-access-r4n2r\") pod \"829a4d78-6c46-448b-af0f-e8d8a7daefd7\" (UID: \"829a4d78-6c46-448b-af0f-e8d8a7daefd7\") " Nov 02 21:13:30.635962 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.635964 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/112e3a47-4817-4237-b296-aa3e260584ef-bundle\" (UniqueName: \"kubernetes.io/empty-dir/112e3a47-4817-4237-b296-aa3e260584ef-bundle\") pod \"112e3a47-4817-4237-b296-aa3e260584ef\" (UID: \"112e3a47-4817-4237-b296-aa3e260584ef\") " Nov 02 21:13:30.636216 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.635994 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/466e29e9-9ab4-42ec-bf83-7c781a15e084-bundle\" (UniqueName: \"kubernetes.io/empty-dir/466e29e9-9ab4-42ec-bf83-7c781a15e084-bundle\") pod \"466e29e9-9ab4-42ec-bf83-7c781a15e084\" (UID: \"466e29e9-9ab4-42ec-bf83-7c781a15e084\") " Nov 02 21:13:30.636216 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.636010 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/112e3a47-4817-4237-b296-aa3e260584ef-util\" (UniqueName: \"kubernetes.io/empty-dir/112e3a47-4817-4237-b296-aa3e260584ef-util\") pod \"112e3a47-4817-4237-b296-aa3e260584ef\" (UID: \"112e3a47-4817-4237-b296-aa3e260584ef\") " Nov 02 21:13:30.636216 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.636027 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/112e3a47-4817-4237-b296-aa3e260584ef-kube-api-access-dcg97\" (UniqueName: \"kubernetes.io/projected/112e3a47-4817-4237-b296-aa3e260584ef-kube-api-access-dcg97\") pod \"112e3a47-4817-4237-b296-aa3e260584ef\" (UID: \"112e3a47-4817-4237-b296-aa3e260584ef\") " Nov 02 21:13:30.636216 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.636090 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/829a4d78-6c46-448b-af0f-e8d8a7daefd7-util\" (UniqueName: \"kubernetes.io/empty-dir/829a4d78-6c46-448b-af0f-e8d8a7daefd7-util\") pod \"829a4d78-6c46-448b-af0f-e8d8a7daefd7\" (UID: \"829a4d78-6c46-448b-af0f-e8d8a7daefd7\") " Nov 02 21:13:30.636216 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.636123 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/829a4d78-6c46-448b-af0f-e8d8a7daefd7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/829a4d78-6c46-448b-af0f-e8d8a7daefd7-bundle\") pod \"829a4d78-6c46-448b-af0f-e8d8a7daefd7\" (UID: \"829a4d78-6c46-448b-af0f-e8d8a7daefd7\") " Nov 02 21:13:30.636465 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.636227 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/466e29e9-9ab4-42ec-bf83-7c781a15e084-kube-api-access-x7m4d\" (UniqueName: \"kubernetes.io/projected/466e29e9-9ab4-42ec-bf83-7c781a15e084-kube-api-access-x7m4d\") pod \"466e29e9-9ab4-42ec-bf83-7c781a15e084\" (UID: \"466e29e9-9ab4-42ec-bf83-7c781a15e084\") " Nov 02 21:13:30.636465 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.636265 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/466e29e9-9ab4-42ec-bf83-7c781a15e084-util\" (UniqueName: \"kubernetes.io/empty-dir/466e29e9-9ab4-42ec-bf83-7c781a15e084-util\") pod \"466e29e9-9ab4-42ec-bf83-7c781a15e084\" (UID: \"466e29e9-9ab4-42ec-bf83-7c781a15e084\") " Nov 02 21:13:30.636563 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.636534 4935 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab8ef755-2409-4114-ac23-31937bb52b4e-util\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:30.636563 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.636556 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q2gw\" (UniqueName: \"kubernetes.io/projected/ab8ef755-2409-4114-ac23-31937bb52b4e-kube-api-access-7q2gw\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:30.636658 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.636570 4935 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab8ef755-2409-4114-ac23-31937bb52b4e-bundle\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:30.637158 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.637112 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/112e3a47-4817-4237-b296-aa3e260584ef-bundle" pod "112e3a47-4817-4237-b296-aa3e260584ef" (UID: "112e3a47-4817-4237-b296-aa3e260584ef"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:30.637272 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.637220 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/466e29e9-9ab4-42ec-bf83-7c781a15e084-bundle" pod "466e29e9-9ab4-42ec-bf83-7c781a15e084" (UID: "466e29e9-9ab4-42ec-bf83-7c781a15e084"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:30.637480 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.637447 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/829a4d78-6c46-448b-af0f-e8d8a7daefd7-bundle" pod "829a4d78-6c46-448b-af0f-e8d8a7daefd7" (UID: "829a4d78-6c46-448b-af0f-e8d8a7daefd7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:30.638558 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.638527 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/112e3a47-4817-4237-b296-aa3e260584ef-kube-api-access-dcg97" pod "112e3a47-4817-4237-b296-aa3e260584ef" (UID: "112e3a47-4817-4237-b296-aa3e260584ef"). InnerVolumeSpecName "kube-api-access-dcg97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:13:30.638671 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.638625 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/829a4d78-6c46-448b-af0f-e8d8a7daefd7-kube-api-access-r4n2r" pod "829a4d78-6c46-448b-af0f-e8d8a7daefd7" (UID: "829a4d78-6c46-448b-af0f-e8d8a7daefd7"). InnerVolumeSpecName "kube-api-access-r4n2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:13:30.639099 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.639080 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/466e29e9-9ab4-42ec-bf83-7c781a15e084-kube-api-access-x7m4d" pod "466e29e9-9ab4-42ec-bf83-7c781a15e084" (UID: "466e29e9-9ab4-42ec-bf83-7c781a15e084"). InnerVolumeSpecName "kube-api-access-x7m4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:13:30.642563 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.642540 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/112e3a47-4817-4237-b296-aa3e260584ef-util" pod "112e3a47-4817-4237-b296-aa3e260584ef" (UID: "112e3a47-4817-4237-b296-aa3e260584ef"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:30.643192 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.643170 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/829a4d78-6c46-448b-af0f-e8d8a7daefd7-util" pod "829a4d78-6c46-448b-af0f-e8d8a7daefd7" (UID: "829a4d78-6c46-448b-af0f-e8d8a7daefd7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:30.643378 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.643357 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/466e29e9-9ab4-42ec-bf83-7c781a15e084-util" pod "466e29e9-9ab4-42ec-bf83-7c781a15e084" (UID: "466e29e9-9ab4-42ec-bf83-7c781a15e084"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:30.737106 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.737075 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7m4d\" (UniqueName: \"kubernetes.io/projected/466e29e9-9ab4-42ec-bf83-7c781a15e084-kube-api-access-x7m4d\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:30.737106 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.737104 4935 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/466e29e9-9ab4-42ec-bf83-7c781a15e084-util\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:30.737106 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.737114 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4n2r\" (UniqueName: \"kubernetes.io/projected/829a4d78-6c46-448b-af0f-e8d8a7daefd7-kube-api-access-r4n2r\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:30.737339 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.737125 4935 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/112e3a47-4817-4237-b296-aa3e260584ef-bundle\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:30.737339 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.737158 4935 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/466e29e9-9ab4-42ec-bf83-7c781a15e084-bundle\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:30.737339 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.737170 4935 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/112e3a47-4817-4237-b296-aa3e260584ef-util\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:30.737339 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.737181 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcg97\" (UniqueName: \"kubernetes.io/projected/112e3a47-4817-4237-b296-aa3e260584ef-kube-api-access-dcg97\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:30.737339 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.737189 4935 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/829a4d78-6c46-448b-af0f-e8d8a7daefd7-util\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:30.737339 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:30.737198 4935 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/829a4d78-6c46-448b-af0f-e8d8a7daefd7-bundle\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:31.121364 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.121316 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:31.279438 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.279409 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8" Nov 02 21:13:31.279854 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.279403 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atx4p8" event={"ID":"466e29e9-9ab4-42ec-bf83-7c781a15e084","Type":"ContainerDied","Data":"ebe2860e4c259431c36eaadf04912dc8267efd58f96c2e50e48fe87309f44237"} Nov 02 21:13:31.279854 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.279565 4935 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebe2860e4c259431c36eaadf04912dc8267efd58f96c2e50e48fe87309f44237" Nov 02 21:13:31.281412 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.281391 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt" Nov 02 21:13:31.281543 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.281419 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fj5kt" event={"ID":"ab8ef755-2409-4114-ac23-31937bb52b4e","Type":"ContainerDied","Data":"da10fb407d843b1cfd506ce174fbdbeb68bab4c2d00eb84480d17b88a2cf842e"} Nov 02 21:13:31.281543 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.281465 4935 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da10fb407d843b1cfd506ce174fbdbeb68bab4c2d00eb84480d17b88a2cf842e" Nov 02 21:13:31.283583 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.283564 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w" Nov 02 21:13:31.283691 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.283584 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c79l6w" event={"ID":"112e3a47-4817-4237-b296-aa3e260584ef","Type":"ContainerDied","Data":"44662897dfb5e38fae0efb3fff75d38be8cfb34aa5fff5adac68fd17242dc688"} Nov 02 21:13:31.283691 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.283622 4935 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44662897dfb5e38fae0efb3fff75d38be8cfb34aa5fff5adac68fd17242dc688" Nov 02 21:13:31.285763 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.285743 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b" Nov 02 21:13:31.285853 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.285762 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71fcg5b" event={"ID":"829a4d78-6c46-448b-af0f-e8d8a7daefd7","Type":"ContainerDied","Data":"492636fff5ecb3d48cb1191ccc58767c4d7530a7ece54cda9d47c8bd7efb7cdd"} Nov 02 21:13:31.285853 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.285814 4935 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="492636fff5ecb3d48cb1191ccc58767c4d7530a7ece54cda9d47c8bd7efb7cdd" Nov 02 21:13:31.447265 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.447154 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj" Nov 02 21:13:31.460785 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.460757 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt" Nov 02 21:13:31.485513 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.485490 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9" Nov 02 21:13:31.494245 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.494225 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh" Nov 02 21:13:31.643972 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.643941 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa46e508-065d-4ce1-b46a-ed1194a6e3cb-util\" (UniqueName: \"kubernetes.io/empty-dir/aa46e508-065d-4ce1-b46a-ed1194a6e3cb-util\") pod \"aa46e508-065d-4ce1-b46a-ed1194a6e3cb\" (UID: \"aa46e508-065d-4ce1-b46a-ed1194a6e3cb\") " Nov 02 21:13:31.644119 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.643983 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa46e508-065d-4ce1-b46a-ed1194a6e3cb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa46e508-065d-4ce1-b46a-ed1194a6e3cb-bundle\") pod \"aa46e508-065d-4ce1-b46a-ed1194a6e3cb\" (UID: \"aa46e508-065d-4ce1-b46a-ed1194a6e3cb\") " Nov 02 21:13:31.644119 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.644017 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c-kube-api-access-6cp69\" (UniqueName: \"kubernetes.io/projected/9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c-kube-api-access-6cp69\") pod \"9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c\" (UID: \"9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c\") " Nov 02 21:13:31.644119 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.644038 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa46e508-065d-4ce1-b46a-ed1194a6e3cb-kube-api-access-pjrzz\" (UniqueName: \"kubernetes.io/projected/aa46e508-065d-4ce1-b46a-ed1194a6e3cb-kube-api-access-pjrzz\") pod \"aa46e508-065d-4ce1-b46a-ed1194a6e3cb\" (UID: \"aa46e508-065d-4ce1-b46a-ed1194a6e3cb\") " Nov 02 21:13:31.644119 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.644060 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeca5d99-bcef-4f43-9991-305e841a9a9e-kube-api-access-6khvm\" (UniqueName: \"kubernetes.io/projected/aeca5d99-bcef-4f43-9991-305e841a9a9e-kube-api-access-6khvm\") pod \"aeca5d99-bcef-4f43-9991-305e841a9a9e\" (UID: \"aeca5d99-bcef-4f43-9991-305e841a9a9e\") " Nov 02 21:13:31.644119 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.644083 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/701c4c49-710a-4b9c-b050-d2ea23303741-util\" (UniqueName: \"kubernetes.io/empty-dir/701c4c49-710a-4b9c-b050-d2ea23303741-util\") pod \"701c4c49-710a-4b9c-b050-d2ea23303741\" (UID: \"701c4c49-710a-4b9c-b050-d2ea23303741\") " Nov 02 21:13:31.644119 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.644109 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aeca5d99-bcef-4f43-9991-305e841a9a9e-util\" (UniqueName: \"kubernetes.io/empty-dir/aeca5d99-bcef-4f43-9991-305e841a9a9e-util\") pod \"aeca5d99-bcef-4f43-9991-305e841a9a9e\" (UID: \"aeca5d99-bcef-4f43-9991-305e841a9a9e\") " Nov 02 21:13:31.644462 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.644170 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/701c4c49-710a-4b9c-b050-d2ea23303741-kube-api-access-kvs6g\" (UniqueName: \"kubernetes.io/projected/701c4c49-710a-4b9c-b050-d2ea23303741-kube-api-access-kvs6g\") pod \"701c4c49-710a-4b9c-b050-d2ea23303741\" (UID: \"701c4c49-710a-4b9c-b050-d2ea23303741\") " Nov 02 21:13:31.644462 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.644208 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c-bundle\") pod \"9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c\" (UID: \"9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c\") " Nov 02 21:13:31.644462 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.644262 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/701c4c49-710a-4b9c-b050-d2ea23303741-bundle\" (UniqueName: \"kubernetes.io/empty-dir/701c4c49-710a-4b9c-b050-d2ea23303741-bundle\") pod \"701c4c49-710a-4b9c-b050-d2ea23303741\" (UID: \"701c4c49-710a-4b9c-b050-d2ea23303741\") " Nov 02 21:13:31.644462 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.644289 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c-util\" (UniqueName: \"kubernetes.io/empty-dir/9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c-util\") pod \"9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c\" (UID: \"9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c\") " Nov 02 21:13:31.644462 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.644338 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aeca5d99-bcef-4f43-9991-305e841a9a9e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/aeca5d99-bcef-4f43-9991-305e841a9a9e-bundle\") pod \"aeca5d99-bcef-4f43-9991-305e841a9a9e\" (UID: \"aeca5d99-bcef-4f43-9991-305e841a9a9e\") " Nov 02 21:13:31.644791 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.644502 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa46e508-065d-4ce1-b46a-ed1194a6e3cb-bundle" pod "aa46e508-065d-4ce1-b46a-ed1194a6e3cb" (UID: "aa46e508-065d-4ce1-b46a-ed1194a6e3cb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:31.644791 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.644641 4935 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa46e508-065d-4ce1-b46a-ed1194a6e3cb-bundle\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:31.645023 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.645001 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c-bundle" pod "9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c" (UID: "9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:31.645565 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.645341 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeca5d99-bcef-4f43-9991-305e841a9a9e-bundle" pod "aeca5d99-bcef-4f43-9991-305e841a9a9e" (UID: "aeca5d99-bcef-4f43-9991-305e841a9a9e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:31.645683 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.645573 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701c4c49-710a-4b9c-b050-d2ea23303741-bundle" pod "701c4c49-710a-4b9c-b050-d2ea23303741" (UID: "701c4c49-710a-4b9c-b050-d2ea23303741"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:31.646984 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.646951 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeca5d99-bcef-4f43-9991-305e841a9a9e-kube-api-access-6khvm" pod "aeca5d99-bcef-4f43-9991-305e841a9a9e" (UID: "aeca5d99-bcef-4f43-9991-305e841a9a9e"). InnerVolumeSpecName "kube-api-access-6khvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:13:31.647608 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.647583 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c-kube-api-access-6cp69" pod "9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c" (UID: "9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c"). InnerVolumeSpecName "kube-api-access-6cp69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:13:31.648075 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.648057 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/701c4c49-710a-4b9c-b050-d2ea23303741-kube-api-access-kvs6g" pod "701c4c49-710a-4b9c-b050-d2ea23303741" (UID: "701c4c49-710a-4b9c-b050-d2ea23303741"). InnerVolumeSpecName "kube-api-access-kvs6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:13:31.648425 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.648407 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa46e508-065d-4ce1-b46a-ed1194a6e3cb-kube-api-access-pjrzz" pod "aa46e508-065d-4ce1-b46a-ed1194a6e3cb" (UID: "aa46e508-065d-4ce1-b46a-ed1194a6e3cb"). InnerVolumeSpecName "kube-api-access-pjrzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:13:31.649942 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.649921 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701c4c49-710a-4b9c-b050-d2ea23303741-util" pod "701c4c49-710a-4b9c-b050-d2ea23303741" (UID: "701c4c49-710a-4b9c-b050-d2ea23303741"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:31.652669 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.652646 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa46e508-065d-4ce1-b46a-ed1194a6e3cb-util" pod "aa46e508-065d-4ce1-b46a-ed1194a6e3cb" (UID: "aa46e508-065d-4ce1-b46a-ed1194a6e3cb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:31.652854 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.652836 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c-util" pod "9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c" (UID: "9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:31.654430 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.654411 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeca5d99-bcef-4f43-9991-305e841a9a9e-util" pod "aeca5d99-bcef-4f43-9991-305e841a9a9e" (UID: "aeca5d99-bcef-4f43-9991-305e841a9a9e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:31.746012 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.745980 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6khvm\" (UniqueName: \"kubernetes.io/projected/aeca5d99-bcef-4f43-9991-305e841a9a9e-kube-api-access-6khvm\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:31.746012 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.746008 4935 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/701c4c49-710a-4b9c-b050-d2ea23303741-util\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:31.746012 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.746017 4935 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aeca5d99-bcef-4f43-9991-305e841a9a9e-util\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:31.746241 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.746026 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvs6g\" (UniqueName: \"kubernetes.io/projected/701c4c49-710a-4b9c-b050-d2ea23303741-kube-api-access-kvs6g\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:31.746241 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.746037 4935 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c-bundle\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:31.746241 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.746046 4935 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/701c4c49-710a-4b9c-b050-d2ea23303741-bundle\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:31.746241 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.746052 4935 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c-util\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:31.746241 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.746060 4935 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aeca5d99-bcef-4f43-9991-305e841a9a9e-bundle\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:31.746241 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.746068 4935 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa46e508-065d-4ce1-b46a-ed1194a6e3cb-util\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:31.746241 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.746075 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cp69\" (UniqueName: \"kubernetes.io/projected/9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c-kube-api-access-6cp69\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:31.746241 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:31.746084 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjrzz\" (UniqueName: \"kubernetes.io/projected/aa46e508-065d-4ce1-b46a-ed1194a6e3cb-kube-api-access-pjrzz\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:32.291909 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:32.291872 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9" Nov 02 21:13:32.291909 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:32.291896 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bq7j9" event={"ID":"9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c","Type":"ContainerDied","Data":"f73b7fa7d640322d4547907ef80a006b4d588b05667910ebd10c21510deef091"} Nov 02 21:13:32.292445 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:32.291953 4935 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f73b7fa7d640322d4547907ef80a006b4d588b05667910ebd10c21510deef091" Nov 02 21:13:32.293927 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:32.293904 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj" Nov 02 21:13:32.294081 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:32.293917 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644j6prj" event={"ID":"aa46e508-065d-4ce1-b46a-ed1194a6e3cb","Type":"ContainerDied","Data":"3420e10f50f5b3335efd055c5fd01b51cf2485acb8757683e9e6c74710e414b9"} Nov 02 21:13:32.294230 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:32.294102 4935 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3420e10f50f5b3335efd055c5fd01b51cf2485acb8757683e9e6c74710e414b9" Nov 02 21:13:32.296146 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:32.296112 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh" Nov 02 21:13:32.296146 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:32.296121 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2wlxh" event={"ID":"701c4c49-710a-4b9c-b050-d2ea23303741","Type":"ContainerDied","Data":"c1c1cb646f41477a64f1fcaa33f8bd2297793aad25149be1c4227380005af066"} Nov 02 21:13:32.296300 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:32.296172 4935 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1c1cb646f41477a64f1fcaa33f8bd2297793aad25149be1c4227380005af066" Nov 02 21:13:32.298261 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:32.298238 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt" event={"ID":"aeca5d99-bcef-4f43-9991-305e841a9a9e","Type":"ContainerDied","Data":"76053cc45d54e72487497984c65d016703b2821d299fd6d9fa63ca76b92f986f"} Nov 02 21:13:32.298355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:32.298279 4935 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76053cc45d54e72487497984c65d016703b2821d299fd6d9fa63ca76b92f986f" Nov 02 21:13:32.298355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:32.298246 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446gpmt" Nov 02 21:13:35.162102 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162070 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xlr8t"] Nov 02 21:13:35.162568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162413 4935 cpu_manager.go:414] "containerMap: removing container" podUID="aa46e508-065d-4ce1-b46a-ed1194a6e3cb" containerName="extract" Nov 02 21:13:35.162568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162425 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa46e508-065d-4ce1-b46a-ed1194a6e3cb" containerName="extract" Nov 02 21:13:35.162568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162435 4935 cpu_manager.go:414] "containerMap: removing container" podUID="701c4c49-710a-4b9c-b050-d2ea23303741" containerName="extract" Nov 02 21:13:35.162568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162440 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="701c4c49-710a-4b9c-b050-d2ea23303741" containerName="extract" Nov 02 21:13:35.162568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162458 4935 cpu_manager.go:414] "containerMap: removing container" podUID="701c4c49-710a-4b9c-b050-d2ea23303741" containerName="util" Nov 02 21:13:35.162568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162463 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="701c4c49-710a-4b9c-b050-d2ea23303741" containerName="util" Nov 02 21:13:35.162568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162470 4935 cpu_manager.go:414] "containerMap: removing container" podUID="aa46e508-065d-4ce1-b46a-ed1194a6e3cb" containerName="util" Nov 02 21:13:35.162568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162475 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa46e508-065d-4ce1-b46a-ed1194a6e3cb" containerName="util" Nov 02 21:13:35.162568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162482 4935 cpu_manager.go:414] "containerMap: removing container" podUID="112e3a47-4817-4237-b296-aa3e260584ef" containerName="pull" Nov 02 21:13:35.162568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162487 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="112e3a47-4817-4237-b296-aa3e260584ef" containerName="pull" Nov 02 21:13:35.162568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162493 4935 cpu_manager.go:414] "containerMap: removing container" podUID="112e3a47-4817-4237-b296-aa3e260584ef" containerName="extract" Nov 02 21:13:35.162568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162497 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="112e3a47-4817-4237-b296-aa3e260584ef" containerName="extract" Nov 02 21:13:35.162568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162505 4935 cpu_manager.go:414] "containerMap: removing container" podUID="829a4d78-6c46-448b-af0f-e8d8a7daefd7" containerName="util" Nov 02 21:13:35.162568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162510 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="829a4d78-6c46-448b-af0f-e8d8a7daefd7" containerName="util" Nov 02 21:13:35.162568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162517 4935 cpu_manager.go:414] "containerMap: removing container" podUID="9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c" containerName="pull" Nov 02 21:13:35.162568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162522 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c" containerName="pull" Nov 02 21:13:35.162568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162532 4935 cpu_manager.go:414] "containerMap: removing container" podUID="ab8ef755-2409-4114-ac23-31937bb52b4e" containerName="util" Nov 02 21:13:35.162568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162537 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab8ef755-2409-4114-ac23-31937bb52b4e" containerName="util" Nov 02 21:13:35.162568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162543 4935 cpu_manager.go:414] "containerMap: removing container" podUID="466e29e9-9ab4-42ec-bf83-7c781a15e084" containerName="util" Nov 02 21:13:35.162568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162547 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="466e29e9-9ab4-42ec-bf83-7c781a15e084" containerName="util" Nov 02 21:13:35.162568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162553 4935 cpu_manager.go:414] "containerMap: removing container" podUID="466e29e9-9ab4-42ec-bf83-7c781a15e084" containerName="pull" Nov 02 21:13:35.162568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162557 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="466e29e9-9ab4-42ec-bf83-7c781a15e084" containerName="pull" Nov 02 21:13:35.162568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162563 4935 cpu_manager.go:414] "containerMap: removing container" podUID="829a4d78-6c46-448b-af0f-e8d8a7daefd7" containerName="pull" Nov 02 21:13:35.162568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162567 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="829a4d78-6c46-448b-af0f-e8d8a7daefd7" containerName="pull" Nov 02 21:13:35.162568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162573 4935 cpu_manager.go:414] "containerMap: removing container" podUID="aeca5d99-bcef-4f43-9991-305e841a9a9e" containerName="pull" Nov 02 21:13:35.162568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162586 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeca5d99-bcef-4f43-9991-305e841a9a9e" containerName="pull" Nov 02 21:13:35.162568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162593 4935 cpu_manager.go:414] "containerMap: removing container" podUID="aa46e508-065d-4ce1-b46a-ed1194a6e3cb" containerName="pull" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162597 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa46e508-065d-4ce1-b46a-ed1194a6e3cb" containerName="pull" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162606 4935 cpu_manager.go:414] "containerMap: removing container" podUID="aeca5d99-bcef-4f43-9991-305e841a9a9e" containerName="extract" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162610 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeca5d99-bcef-4f43-9991-305e841a9a9e" containerName="extract" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162616 4935 cpu_manager.go:414] "containerMap: removing container" podUID="112e3a47-4817-4237-b296-aa3e260584ef" containerName="util" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162621 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="112e3a47-4817-4237-b296-aa3e260584ef" containerName="util" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162628 4935 cpu_manager.go:414] "containerMap: removing container" podUID="9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c" containerName="util" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162632 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c" containerName="util" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162639 4935 cpu_manager.go:414] "containerMap: removing container" podUID="701c4c49-710a-4b9c-b050-d2ea23303741" containerName="pull" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162642 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="701c4c49-710a-4b9c-b050-d2ea23303741" containerName="pull" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162648 4935 cpu_manager.go:414] "containerMap: removing container" podUID="829a4d78-6c46-448b-af0f-e8d8a7daefd7" containerName="extract" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162652 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="829a4d78-6c46-448b-af0f-e8d8a7daefd7" containerName="extract" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162657 4935 cpu_manager.go:414] "containerMap: removing container" podUID="ab8ef755-2409-4114-ac23-31937bb52b4e" containerName="extract" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162661 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab8ef755-2409-4114-ac23-31937bb52b4e" containerName="extract" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162667 4935 cpu_manager.go:414] "containerMap: removing container" podUID="ab8ef755-2409-4114-ac23-31937bb52b4e" containerName="pull" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162671 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab8ef755-2409-4114-ac23-31937bb52b4e" containerName="pull" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162677 4935 cpu_manager.go:414] "containerMap: removing container" podUID="9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c" containerName="extract" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162681 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c" containerName="extract" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162688 4935 cpu_manager.go:414] "containerMap: removing container" podUID="aeca5d99-bcef-4f43-9991-305e841a9a9e" containerName="util" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162691 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeca5d99-bcef-4f43-9991-305e841a9a9e" containerName="util" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162696 4935 cpu_manager.go:414] "containerMap: removing container" podUID="466e29e9-9ab4-42ec-bf83-7c781a15e084" containerName="extract" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162701 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="466e29e9-9ab4-42ec-bf83-7c781a15e084" containerName="extract" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162744 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeca5d99-bcef-4f43-9991-305e841a9a9e" containerName="extract" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162751 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="829a4d78-6c46-448b-af0f-e8d8a7daefd7" containerName="extract" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162756 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab8ef755-2409-4114-ac23-31937bb52b4e" containerName="extract" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162762 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ef41d0c-1220-4c93-959b-0c4b7e1b5e9c" containerName="extract" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162768 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="466e29e9-9ab4-42ec-bf83-7c781a15e084" containerName="extract" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162774 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa46e508-065d-4ce1-b46a-ed1194a6e3cb" containerName="extract" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162779 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="701c4c49-710a-4b9c-b050-d2ea23303741" containerName="extract" Nov 02 21:13:35.163355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.162786 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="112e3a47-4817-4237-b296-aa3e260584ef" containerName="extract" Nov 02 21:13:35.168660 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.168641 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xlr8t" Nov 02 21:13:35.171612 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.171592 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l28r9\" (UniqueName: \"kubernetes.io/projected/449ff8c5-c97f-42c9-8aef-93392fbd2f37-kube-api-access-l28r9\") pod \"kuadrant-operator-controller-manager-cf8469d7b-xlr8t\" (UID: \"449ff8c5-c97f-42c9-8aef-93392fbd2f37\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xlr8t" Nov 02 21:13:35.171718 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.171620 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/449ff8c5-c97f-42c9-8aef-93392fbd2f37-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-xlr8t\" (UID: \"449ff8c5-c97f-42c9-8aef-93392fbd2f37\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xlr8t" Nov 02 21:13:35.217705 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.217677 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 21:13:35.217900 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.217884 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 21:13:35.217974 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.217904 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-z8k9l\"" Nov 02 21:13:35.236861 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.236833 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xlr8t"] Nov 02 21:13:35.272541 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.272510 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l28r9\" (UniqueName: \"kubernetes.io/projected/449ff8c5-c97f-42c9-8aef-93392fbd2f37-kube-api-access-l28r9\") pod \"kuadrant-operator-controller-manager-cf8469d7b-xlr8t\" (UID: \"449ff8c5-c97f-42c9-8aef-93392fbd2f37\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xlr8t" Nov 02 21:13:35.272674 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.272550 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/449ff8c5-c97f-42c9-8aef-93392fbd2f37-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-xlr8t\" (UID: \"449ff8c5-c97f-42c9-8aef-93392fbd2f37\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xlr8t" Nov 02 21:13:35.272942 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.272918 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/449ff8c5-c97f-42c9-8aef-93392fbd2f37-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-xlr8t\" (UID: \"449ff8c5-c97f-42c9-8aef-93392fbd2f37\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xlr8t" Nov 02 21:13:35.328650 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.328619 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l28r9\" (UniqueName: \"kubernetes.io/projected/449ff8c5-c97f-42c9-8aef-93392fbd2f37-kube-api-access-l28r9\") pod \"kuadrant-operator-controller-manager-cf8469d7b-xlr8t\" (UID: \"449ff8c5-c97f-42c9-8aef-93392fbd2f37\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xlr8t" Nov 02 21:13:35.482003 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.481907 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xlr8t" Nov 02 21:13:35.890071 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.890038 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:35.957478 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.957445 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xlr8t"] Nov 02 21:13:35.986835 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:13:35.986807 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod449ff8c5_c97f_42c9_8aef_93392fbd2f37.slice/crio-1400d5e551286512f0985f83bf1258cea39dfa59e3dd491533611dd184b5223f WatchSource:0}: Error finding container 1400d5e551286512f0985f83bf1258cea39dfa59e3dd491533611dd184b5223f: Status 404 returned error can't find the container with id 1400d5e551286512f0985f83bf1258cea39dfa59e3dd491533611dd184b5223f Nov 02 21:13:35.989473 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:35.989455 4935 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:13:36.120376 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:36.120342 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:36.317495 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:36.317460 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xlr8t" event={"ID":"449ff8c5-c97f-42c9-8aef-93392fbd2f37","Type":"ContainerStarted","Data":"1400d5e551286512f0985f83bf1258cea39dfa59e3dd491533611dd184b5223f"} Nov 02 21:13:36.471987 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:36.471953 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:36.774706 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:36.774672 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:39.694916 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:39.694823 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-gh5gd"] Nov 02 21:13:39.705497 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:39.705466 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gh5gd" Nov 02 21:13:39.709404 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:39.709276 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gmn8\" (UniqueName: \"kubernetes.io/projected/42f901f6-b299-4cc4-8871-2753cb497f94-kube-api-access-7gmn8\") pod \"authorino-operator-7b7bb4bc77-gh5gd\" (UID: \"42f901f6-b299-4cc4-8871-2753cb497f94\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-gh5gd" Nov 02 21:13:39.807328 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:39.807286 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-gh5gd"] Nov 02 21:13:39.807506 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:39.807444 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-g2pxd\"" Nov 02 21:13:39.810079 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:39.810045 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gmn8\" (UniqueName: \"kubernetes.io/projected/42f901f6-b299-4cc4-8871-2753cb497f94-kube-api-access-7gmn8\") pod \"authorino-operator-7b7bb4bc77-gh5gd\" (UID: \"42f901f6-b299-4cc4-8871-2753cb497f94\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-gh5gd" Nov 02 21:13:39.867986 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:39.867952 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gmn8\" (UniqueName: \"kubernetes.io/projected/42f901f6-b299-4cc4-8871-2753cb497f94-kube-api-access-7gmn8\") pod \"authorino-operator-7b7bb4bc77-gh5gd\" (UID: \"42f901f6-b299-4cc4-8871-2753cb497f94\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-gh5gd" Nov 02 21:13:40.022776 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:40.022743 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gh5gd" Nov 02 21:13:40.399265 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:40.399189 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-gh5gd"] Nov 02 21:13:40.403317 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:13:40.403286 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f901f6_b299_4cc4_8871_2753cb497f94.slice/crio-b2c1b37844d19da64f3f99d36de5e2bd673e05e4b5b5d7508cea2f3233105e07 WatchSource:0}: Error finding container b2c1b37844d19da64f3f99d36de5e2bd673e05e4b5b5d7508cea2f3233105e07: Status 404 returned error can't find the container with id b2c1b37844d19da64f3f99d36de5e2bd673e05e4b5b5d7508cea2f3233105e07 Nov 02 21:13:41.077024 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:41.076989 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:41.298124 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:41.298004 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:41.347711 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:41.347629 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gh5gd" event={"ID":"42f901f6-b299-4cc4-8871-2753cb497f94","Type":"ContainerStarted","Data":"b2c1b37844d19da64f3f99d36de5e2bd673e05e4b5b5d7508cea2f3233105e07"} Nov 02 21:13:41.506936 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:41.506896 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:42.355672 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:42.355637 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xlr8t" event={"ID":"449ff8c5-c97f-42c9-8aef-93392fbd2f37","Type":"ContainerStarted","Data":"d63bd0c5cd68528ccfdad6180859b5c53636e52a2c1224c51002bb7ebf37423b"} Nov 02 21:13:42.356154 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:42.355768 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xlr8t" Nov 02 21:13:42.568454 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:42.568400 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xlr8t" podStartSLOduration=2.009601466 podStartE2EDuration="7.568379678s" totalImagesPullingTime="5.558778212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:13:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:42.52026727 +0000 UTC m=+314.959841992" watchObservedRunningTime="2026-11-02 21:13:42.568379678 +0000 UTC m=+315.007954397" Nov 02 21:13:43.361772 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:43.361668 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gh5gd" event={"ID":"42f901f6-b299-4cc4-8871-2753cb497f94","Type":"ContainerStarted","Data":"4d6c1214b0f2ac5d18ede4c581a4370eee2415403a7a12c2ce52b6e27be4e345"} Nov 02 21:13:43.361772 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:43.361748 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gh5gd" Nov 02 21:13:43.469480 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:43.469423 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gh5gd" podStartSLOduration=1.861608583 podStartE2EDuration="4.46940855s" totalImagesPullingTime="2.607799967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:13:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:43.468245123 +0000 UTC m=+315.907819844" watchObservedRunningTime="2026-11-02 21:13:43.46940855 +0000 UTC m=+315.908983273" Nov 02 21:13:45.309883 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:45.309849 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:45.542052 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:45.542016 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:46.819372 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:46.819336 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:47.074091 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:47.074012 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:47.293000 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:47.292969 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:47.818661 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:47.818631 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:48.036147 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:48.036112 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:48.382059 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:48.382027 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-8ggc5"] Nov 02 21:13:48.385636 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:48.385614 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8ggc5" Nov 02 21:13:48.484479 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:48.484445 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4djgp\" (UniqueName: \"kubernetes.io/projected/7797919e-d037-4b9c-9eb9-3d0d8af052fb-kube-api-access-4djgp\") pod \"limitador-operator-controller-manager-5778bb4449-8ggc5\" (UID: \"7797919e-d037-4b9c-9eb9-3d0d8af052fb\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8ggc5" Nov 02 21:13:48.486034 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:48.486009 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-rs5bh\"" Nov 02 21:13:48.487753 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:48.487728 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-8ggc5"] Nov 02 21:13:48.585475 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:48.585442 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4djgp\" (UniqueName: \"kubernetes.io/projected/7797919e-d037-4b9c-9eb9-3d0d8af052fb-kube-api-access-4djgp\") pod \"limitador-operator-controller-manager-5778bb4449-8ggc5\" (UID: \"7797919e-d037-4b9c-9eb9-3d0d8af052fb\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8ggc5" Nov 02 21:13:48.642549 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:48.642477 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4djgp\" (UniqueName: \"kubernetes.io/projected/7797919e-d037-4b9c-9eb9-3d0d8af052fb-kube-api-access-4djgp\") pod \"limitador-operator-controller-manager-5778bb4449-8ggc5\" (UID: \"7797919e-d037-4b9c-9eb9-3d0d8af052fb\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8ggc5" Nov 02 21:13:48.698117 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:48.698088 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8ggc5" Nov 02 21:13:49.060733 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:49.060697 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-8ggc5"] Nov 02 21:13:49.063950 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:13:49.063919 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7797919e_d037_4b9c_9eb9_3d0d8af052fb.slice/crio-86a683521df98fe3bb6165d07608588acb2ffcce86db97446262bf524b7c5255 WatchSource:0}: Error finding container 86a683521df98fe3bb6165d07608588acb2ffcce86db97446262bf524b7c5255: Status 404 returned error can't find the container with id 86a683521df98fe3bb6165d07608588acb2ffcce86db97446262bf524b7c5255 Nov 02 21:13:49.388661 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:49.388578 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8ggc5" event={"ID":"7797919e-d037-4b9c-9eb9-3d0d8af052fb","Type":"ContainerStarted","Data":"86a683521df98fe3bb6165d07608588acb2ffcce86db97446262bf524b7c5255"} Nov 02 21:13:49.951087 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:49.951054 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:50.174174 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:50.174121 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:50.401501 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:50.401471 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:50.605984 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:50.605951 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:50.967521 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:50.967484 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:51.502699 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.502664 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:51.654441 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.654401 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v"] Nov 02 21:13:51.658779 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.658752 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v" Nov 02 21:13:51.710675 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.710506 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 21:13:51.710675 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.710546 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 21:13:51.710675 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.710587 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 21:13:51.710675 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.710639 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 21:13:51.710966 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.710886 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d56qr\" (UniqueName: \"kubernetes.io/projected/32a3abf1-5f8c-4137-90cb-395896cd2178-kube-api-access-d56qr\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-9vj7v\" (UID: \"32a3abf1-5f8c-4137-90cb-395896cd2178\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v" Nov 02 21:13:51.710966 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.710928 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32a3abf1-5f8c-4137-90cb-395896cd2178-webhook-cert\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-9vj7v\" (UID: \"32a3abf1-5f8c-4137-90cb-395896cd2178\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v" Nov 02 21:13:51.711077 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.710970 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/32a3abf1-5f8c-4137-90cb-395896cd2178-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-9vj7v\" (UID: \"32a3abf1-5f8c-4137-90cb-395896cd2178\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v" Nov 02 21:13:51.711167 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.711121 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/32a3abf1-5f8c-4137-90cb-395896cd2178-metrics-cert\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-9vj7v\" (UID: \"32a3abf1-5f8c-4137-90cb-395896cd2178\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v" Nov 02 21:13:51.711257 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.711231 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32a3abf1-5f8c-4137-90cb-395896cd2178-tmp\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-9vj7v\" (UID: \"32a3abf1-5f8c-4137-90cb-395896cd2178\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v" Nov 02 21:13:51.711334 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.711306 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/32a3abf1-5f8c-4137-90cb-395896cd2178-manifests\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-9vj7v\" (UID: \"32a3abf1-5f8c-4137-90cb-395896cd2178\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v" Nov 02 21:13:51.748324 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.748282 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:51.756996 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.756915 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-rhszk\"" Nov 02 21:13:51.757150 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.757005 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 21:13:51.800587 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.800555 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v"] Nov 02 21:13:51.812159 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.812114 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32a3abf1-5f8c-4137-90cb-395896cd2178-tmp\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-9vj7v\" (UID: \"32a3abf1-5f8c-4137-90cb-395896cd2178\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v" Nov 02 21:13:51.812299 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.812189 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/32a3abf1-5f8c-4137-90cb-395896cd2178-manifests\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-9vj7v\" (UID: \"32a3abf1-5f8c-4137-90cb-395896cd2178\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v" Nov 02 21:13:51.812299 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.812230 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d56qr\" (UniqueName: \"kubernetes.io/projected/32a3abf1-5f8c-4137-90cb-395896cd2178-kube-api-access-d56qr\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-9vj7v\" (UID: \"32a3abf1-5f8c-4137-90cb-395896cd2178\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v" Nov 02 21:13:51.812299 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.812259 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32a3abf1-5f8c-4137-90cb-395896cd2178-webhook-cert\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-9vj7v\" (UID: \"32a3abf1-5f8c-4137-90cb-395896cd2178\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v" Nov 02 21:13:51.812299 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.812294 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/32a3abf1-5f8c-4137-90cb-395896cd2178-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-9vj7v\" (UID: \"32a3abf1-5f8c-4137-90cb-395896cd2178\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v" Nov 02 21:13:51.812500 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.812349 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/32a3abf1-5f8c-4137-90cb-395896cd2178-metrics-cert\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-9vj7v\" (UID: \"32a3abf1-5f8c-4137-90cb-395896cd2178\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v" Nov 02 21:13:51.812677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.812630 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/32a3abf1-5f8c-4137-90cb-395896cd2178-manifests\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-9vj7v\" (UID: \"32a3abf1-5f8c-4137-90cb-395896cd2178\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v" Nov 02 21:13:51.814445 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.814414 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32a3abf1-5f8c-4137-90cb-395896cd2178-tmp\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-9vj7v\" (UID: \"32a3abf1-5f8c-4137-90cb-395896cd2178\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v" Nov 02 21:13:51.814824 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.814800 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/32a3abf1-5f8c-4137-90cb-395896cd2178-metrics-cert\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-9vj7v\" (UID: \"32a3abf1-5f8c-4137-90cb-395896cd2178\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v" Nov 02 21:13:51.814927 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.814885 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32a3abf1-5f8c-4137-90cb-395896cd2178-webhook-cert\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-9vj7v\" (UID: \"32a3abf1-5f8c-4137-90cb-395896cd2178\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v" Nov 02 21:13:51.815647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.815621 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/32a3abf1-5f8c-4137-90cb-395896cd2178-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-9vj7v\" (UID: \"32a3abf1-5f8c-4137-90cb-395896cd2178\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v" Nov 02 21:13:51.869401 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.869373 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d56qr\" (UniqueName: \"kubernetes.io/projected/32a3abf1-5f8c-4137-90cb-395896cd2178-kube-api-access-d56qr\") pod \"opendatahub-operator-controller-manager-7655d4cbf5-9vj7v\" (UID: \"32a3abf1-5f8c-4137-90cb-395896cd2178\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v" Nov 02 21:13:51.976405 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.976371 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v" Nov 02 21:13:51.992361 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:51.992330 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:52.229400 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:52.229366 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:52.406501 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:52.406411 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8ggc5" event={"ID":"7797919e-d037-4b9c-9eb9-3d0d8af052fb","Type":"ContainerStarted","Data":"f91f3c477ad394780a72e02e66fbabcc7b9cb8dcf85b91c8e2954f3ab661f753"} Nov 02 21:13:52.406679 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:52.406533 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8ggc5" Nov 02 21:13:52.428913 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:52.428887 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:52.451809 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:52.451790 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v"] Nov 02 21:13:52.453804 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:13:52.453782 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a3abf1_5f8c_4137_90cb_395896cd2178.slice/crio-0e25c0a572b543e2a08b6e5a1afa3d7ced94a7c1fdc9362041899b50cda2ca3d WatchSource:0}: Error finding container 0e25c0a572b543e2a08b6e5a1afa3d7ced94a7c1fdc9362041899b50cda2ca3d: Status 404 returned error can't find the container with id 0e25c0a572b543e2a08b6e5a1afa3d7ced94a7c1fdc9362041899b50cda2ca3d Nov 02 21:13:52.518254 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:52.518194 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8ggc5" podStartSLOduration=1.724771269 podStartE2EDuration="4.51817535s" totalImagesPullingTime="2.793404081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:13:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:52.516417681 +0000 UTC m=+324.955992403" watchObservedRunningTime="2026-11-02 21:13:52.51817535 +0000 UTC m=+324.957750072" Nov 02 21:13:53.364558 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:53.364531 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xlr8t" Nov 02 21:13:53.412948 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:53.412911 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v" event={"ID":"32a3abf1-5f8c-4137-90cb-395896cd2178","Type":"ContainerStarted","Data":"0e25c0a572b543e2a08b6e5a1afa3d7ced94a7c1fdc9362041899b50cda2ca3d"} Nov 02 21:13:54.368802 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:54.368767 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gh5gd" Nov 02 21:13:54.546218 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:54.546176 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xbm86"] Nov 02 21:13:54.552967 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:54.552943 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xbm86" Nov 02 21:13:54.638942 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:54.638858 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88838f5d-567a-4e5e-9dc9-eff4470e53ff-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-xbm86\" (UID: \"88838f5d-567a-4e5e-9dc9-eff4470e53ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xbm86" Nov 02 21:13:54.638942 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:54.638916 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctbgb\" (UniqueName: \"kubernetes.io/projected/88838f5d-567a-4e5e-9dc9-eff4470e53ff-kube-api-access-ctbgb\") pod \"kuadrant-operator-controller-manager-678d87766f-xbm86\" (UID: \"88838f5d-567a-4e5e-9dc9-eff4470e53ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xbm86" Nov 02 21:13:54.650949 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:54.650913 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xbm86"] Nov 02 21:13:54.739811 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:54.739768 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88838f5d-567a-4e5e-9dc9-eff4470e53ff-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-xbm86\" (UID: \"88838f5d-567a-4e5e-9dc9-eff4470e53ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xbm86" Nov 02 21:13:54.739811 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:54.739810 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctbgb\" (UniqueName: \"kubernetes.io/projected/88838f5d-567a-4e5e-9dc9-eff4470e53ff-kube-api-access-ctbgb\") pod \"kuadrant-operator-controller-manager-678d87766f-xbm86\" (UID: \"88838f5d-567a-4e5e-9dc9-eff4470e53ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xbm86" Nov 02 21:13:54.740201 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:54.740173 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88838f5d-567a-4e5e-9dc9-eff4470e53ff-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-xbm86\" (UID: \"88838f5d-567a-4e5e-9dc9-eff4470e53ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xbm86" Nov 02 21:13:54.795106 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:54.795071 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctbgb\" (UniqueName: \"kubernetes.io/projected/88838f5d-567a-4e5e-9dc9-eff4470e53ff-kube-api-access-ctbgb\") pod \"kuadrant-operator-controller-manager-678d87766f-xbm86\" (UID: \"88838f5d-567a-4e5e-9dc9-eff4470e53ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xbm86" Nov 02 21:13:54.870834 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:54.870796 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xbm86" Nov 02 21:13:55.317322 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:55.317282 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xbm86"] Nov 02 21:13:55.323430 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:13:55.323399 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88838f5d_567a_4e5e_9dc9_eff4470e53ff.slice/crio-dfc1ec9750966c1939e5a3c00d24eec3af9a55f089e85e622f0f2604081ad857 WatchSource:0}: Error finding container dfc1ec9750966c1939e5a3c00d24eec3af9a55f089e85e622f0f2604081ad857: Status 404 returned error can't find the container with id dfc1ec9750966c1939e5a3c00d24eec3af9a55f089e85e622f0f2604081ad857 Nov 02 21:13:55.426857 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:55.426818 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xbm86" event={"ID":"88838f5d-567a-4e5e-9dc9-eff4470e53ff","Type":"ContainerStarted","Data":"29ae623bd265182e771ecdb3fa329e750c8c19460a94d965edd5e5987dd488b5"} Nov 02 21:13:55.427247 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:55.426879 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xbm86" event={"ID":"88838f5d-567a-4e5e-9dc9-eff4470e53ff","Type":"ContainerStarted","Data":"dfc1ec9750966c1939e5a3c00d24eec3af9a55f089e85e622f0f2604081ad857"} Nov 02 21:13:55.427247 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:55.426949 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xbm86" Nov 02 21:13:55.543702 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:55.543649 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xbm86" podStartSLOduration=1.5436349759999999 podStartE2EDuration="1.543634976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:55.541769462 +0000 UTC m=+327.981344184" watchObservedRunningTime="2026-11-02 21:13:55.543634976 +0000 UTC m=+327.983209699" Nov 02 21:13:55.660584 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:55.660503 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:55.959509 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:55.959427 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:56.433729 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:56.433690 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v" event={"ID":"32a3abf1-5f8c-4137-90cb-395896cd2178","Type":"ContainerStarted","Data":"2b5362876b91aa894171bd70b22b7ed9e0af5b845a16e8975d009fcdf19adcf5"} Nov 02 21:13:57.440448 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:57.440411 4935 generic.go:356] "Generic (PLEG): container finished" podID="32a3abf1-5f8c-4137-90cb-395896cd2178" containerID="2b5362876b91aa894171bd70b22b7ed9e0af5b845a16e8975d009fcdf19adcf5" exitCode=0 Nov 02 21:13:57.440821 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:57.440499 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v" event={"ID":"32a3abf1-5f8c-4137-90cb-395896cd2178","Type":"ContainerDied","Data":"2b5362876b91aa894171bd70b22b7ed9e0af5b845a16e8975d009fcdf19adcf5"} Nov 02 21:13:57.959332 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:57.959295 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:58.162501 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:58.162463 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:58.447968 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:58.447927 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v" event={"ID":"32a3abf1-5f8c-4137-90cb-395896cd2178","Type":"ContainerStarted","Data":"e2e1ab067b3641860d67c5be62f55a768e86a392bb1cbd1cc1a549b824646f07"} Nov 02 21:13:58.448361 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:58.448076 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v" Nov 02 21:13:58.470179 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:13:58.470125 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:13:58.493505 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:58.493481 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:58.620852 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:58.620755 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v" podStartSLOduration=2.503105105 podStartE2EDuration="7.620739922s" totalImagesPullingTime="4.692343774s" totalInitContainerRuntime="425.291043ms" isStatefulPod=false podCreationTimestamp="2026-11-02 21:13:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:58.61832065 +0000 UTC m=+331.057895372" watchObservedRunningTime="2026-11-02 21:13:58.620739922 +0000 UTC m=+331.060314643" Nov 02 21:13:58.864548 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:58.864508 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:59.071770 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:59.071722 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:59.318158 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:59.318099 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:59.685191 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:59.685148 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:13:59.920599 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:13:59.920564 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:03.415147 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:03.415100 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8ggc5" Nov 02 21:14:05.386877 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:05.386834 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:05.601291 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:05.601240 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:06.436421 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:06.436386 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xbm86" Nov 02 21:14:06.642564 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:06.642521 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xlr8t"] Nov 02 21:14:06.642825 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:06.642772 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xlr8t" podUID="449ff8c5-c97f-42c9-8aef-93392fbd2f37" containerName="manager" containerID="cri-o://d63bd0c5cd68528ccfdad6180859b5c53636e52a2c1224c51002bb7ebf37423b" gracePeriod=10 Nov 02 21:14:06.895671 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:06.895646 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xlr8t" Nov 02 21:14:06.943663 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:06.943635 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/449ff8c5-c97f-42c9-8aef-93392fbd2f37-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/449ff8c5-c97f-42c9-8aef-93392fbd2f37-extensions-socket-volume\") pod \"449ff8c5-c97f-42c9-8aef-93392fbd2f37\" (UID: \"449ff8c5-c97f-42c9-8aef-93392fbd2f37\") " Nov 02 21:14:06.943819 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:06.943720 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/449ff8c5-c97f-42c9-8aef-93392fbd2f37-kube-api-access-l28r9\" (UniqueName: \"kubernetes.io/projected/449ff8c5-c97f-42c9-8aef-93392fbd2f37-kube-api-access-l28r9\") pod \"449ff8c5-c97f-42c9-8aef-93392fbd2f37\" (UID: \"449ff8c5-c97f-42c9-8aef-93392fbd2f37\") " Nov 02 21:14:06.944187 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:06.944163 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449ff8c5-c97f-42c9-8aef-93392fbd2f37-extensions-socket-volume" pod "449ff8c5-c97f-42c9-8aef-93392fbd2f37" (UID: "449ff8c5-c97f-42c9-8aef-93392fbd2f37"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:06.945854 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:06.945832 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/449ff8c5-c97f-42c9-8aef-93392fbd2f37-kube-api-access-l28r9" pod "449ff8c5-c97f-42c9-8aef-93392fbd2f37" (UID: "449ff8c5-c97f-42c9-8aef-93392fbd2f37"). InnerVolumeSpecName "kube-api-access-l28r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:14:07.044408 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:07.044376 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l28r9\" (UniqueName: \"kubernetes.io/projected/449ff8c5-c97f-42c9-8aef-93392fbd2f37-kube-api-access-l28r9\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:07.044408 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:07.044404 4935 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/449ff8c5-c97f-42c9-8aef-93392fbd2f37-extensions-socket-volume\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:07.491278 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:07.491194 4935 generic.go:356] "Generic (PLEG): container finished" podID="449ff8c5-c97f-42c9-8aef-93392fbd2f37" containerID="d63bd0c5cd68528ccfdad6180859b5c53636e52a2c1224c51002bb7ebf37423b" exitCode=0 Nov 02 21:14:07.491278 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:07.491243 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xlr8t" Nov 02 21:14:07.491731 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:07.491280 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xlr8t" event={"ID":"449ff8c5-c97f-42c9-8aef-93392fbd2f37","Type":"ContainerDied","Data":"d63bd0c5cd68528ccfdad6180859b5c53636e52a2c1224c51002bb7ebf37423b"} Nov 02 21:14:07.491731 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:07.491343 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xlr8t" event={"ID":"449ff8c5-c97f-42c9-8aef-93392fbd2f37","Type":"ContainerDied","Data":"1400d5e551286512f0985f83bf1258cea39dfa59e3dd491533611dd184b5223f"} Nov 02 21:14:07.491731 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:07.491366 4935 scope.go:122] "RemoveContainer" containerID="d63bd0c5cd68528ccfdad6180859b5c53636e52a2c1224c51002bb7ebf37423b" Nov 02 21:14:07.501405 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:07.501386 4935 scope.go:122] "RemoveContainer" containerID="d63bd0c5cd68528ccfdad6180859b5c53636e52a2c1224c51002bb7ebf37423b" Nov 02 21:14:07.501676 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:14:07.501657 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d63bd0c5cd68528ccfdad6180859b5c53636e52a2c1224c51002bb7ebf37423b\": container with ID starting with d63bd0c5cd68528ccfdad6180859b5c53636e52a2c1224c51002bb7ebf37423b not found: ID does not exist" containerID="d63bd0c5cd68528ccfdad6180859b5c53636e52a2c1224c51002bb7ebf37423b" Nov 02 21:14:07.501745 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:07.501687 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d63bd0c5cd68528ccfdad6180859b5c53636e52a2c1224c51002bb7ebf37423b"} err="failed to get container status \"d63bd0c5cd68528ccfdad6180859b5c53636e52a2c1224c51002bb7ebf37423b\": rpc error: code = NotFound desc = could not find container \"d63bd0c5cd68528ccfdad6180859b5c53636e52a2c1224c51002bb7ebf37423b\": container with ID starting with d63bd0c5cd68528ccfdad6180859b5c53636e52a2c1224c51002bb7ebf37423b not found: ID does not exist" Nov 02 21:14:07.653034 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:07.653008 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xlr8t"] Nov 02 21:14:07.699440 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:07.699409 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-xlr8t"] Nov 02 21:14:07.774446 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:07.774418 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="449ff8c5-c97f-42c9-8aef-93392fbd2f37" path="/var/lib/kubelet/pods/449ff8c5-c97f-42c9-8aef-93392fbd2f37/volumes" Nov 02 21:14:08.863837 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:08.863800 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:09.104498 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:09.104459 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:09.343729 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:09.343694 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:09.454741 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:09.454716 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7655d4cbf5-9vj7v" Nov 02 21:14:10.690449 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:10.690384 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:10.941676 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:10.941594 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:15.317080 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:15.317049 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:15.515435 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:15.515392 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:19.329952 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:19.329917 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:21.260604 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:21.260560 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:21.478179 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:21.478122 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:21.695801 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:21.695737 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:23.204147 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.204089 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:23.408855 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.408819 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:23.674694 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.674655 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:23.692055 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.692021 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g"] Nov 02 21:14:23.692462 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.692443 4935 cpu_manager.go:414] "containerMap: removing container" podUID="449ff8c5-c97f-42c9-8aef-93392fbd2f37" containerName="manager" Nov 02 21:14:23.692544 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.692463 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="449ff8c5-c97f-42c9-8aef-93392fbd2f37" containerName="manager" Nov 02 21:14:23.692601 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.692564 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="449ff8c5-c97f-42c9-8aef-93392fbd2f37" containerName="manager" Nov 02 21:14:23.696900 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.696879 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:23.782124 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.782088 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52ce2cb-6893-4347-b3aa-6268aae6ceda-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cvz5g\" (UID: \"d52ce2cb-6893-4347-b3aa-6268aae6ceda\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:23.782301 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.782221 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnk9j\" (UniqueName: \"kubernetes.io/projected/d52ce2cb-6893-4347-b3aa-6268aae6ceda-kube-api-access-tnk9j\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cvz5g\" (UID: \"d52ce2cb-6893-4347-b3aa-6268aae6ceda\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:23.782301 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.782255 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52ce2cb-6893-4347-b3aa-6268aae6ceda-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cvz5g\" (UID: \"d52ce2cb-6893-4347-b3aa-6268aae6ceda\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:23.782301 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.782279 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52ce2cb-6893-4347-b3aa-6268aae6ceda-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cvz5g\" (UID: \"d52ce2cb-6893-4347-b3aa-6268aae6ceda\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:23.782523 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.782311 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52ce2cb-6893-4347-b3aa-6268aae6ceda-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cvz5g\" (UID: \"d52ce2cb-6893-4347-b3aa-6268aae6ceda\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:23.782523 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.782443 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52ce2cb-6893-4347-b3aa-6268aae6ceda-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cvz5g\" (UID: \"d52ce2cb-6893-4347-b3aa-6268aae6ceda\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:23.782523 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.782483 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52ce2cb-6893-4347-b3aa-6268aae6ceda-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cvz5g\" (UID: \"d52ce2cb-6893-4347-b3aa-6268aae6ceda\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:23.782662 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.782546 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52ce2cb-6893-4347-b3aa-6268aae6ceda-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cvz5g\" (UID: \"d52ce2cb-6893-4347-b3aa-6268aae6ceda\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:23.782662 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.782569 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-gxlhb\"" Nov 02 21:14:23.782662 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.782610 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52ce2cb-6893-4347-b3aa-6268aae6ceda-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cvz5g\" (UID: \"d52ce2cb-6893-4347-b3aa-6268aae6ceda\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:23.784654 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.784634 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g"] Nov 02 21:14:23.883350 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.883313 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52ce2cb-6893-4347-b3aa-6268aae6ceda-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cvz5g\" (UID: \"d52ce2cb-6893-4347-b3aa-6268aae6ceda\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:23.883350 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.883349 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52ce2cb-6893-4347-b3aa-6268aae6ceda-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cvz5g\" (UID: \"d52ce2cb-6893-4347-b3aa-6268aae6ceda\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:23.883590 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.883387 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52ce2cb-6893-4347-b3aa-6268aae6ceda-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cvz5g\" (UID: \"d52ce2cb-6893-4347-b3aa-6268aae6ceda\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:23.883590 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.883407 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52ce2cb-6893-4347-b3aa-6268aae6ceda-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cvz5g\" (UID: \"d52ce2cb-6893-4347-b3aa-6268aae6ceda\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:23.883590 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.883425 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52ce2cb-6893-4347-b3aa-6268aae6ceda-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cvz5g\" (UID: \"d52ce2cb-6893-4347-b3aa-6268aae6ceda\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:23.883590 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.883468 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnk9j\" (UniqueName: \"kubernetes.io/projected/d52ce2cb-6893-4347-b3aa-6268aae6ceda-kube-api-access-tnk9j\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cvz5g\" (UID: \"d52ce2cb-6893-4347-b3aa-6268aae6ceda\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:23.883590 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.883523 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52ce2cb-6893-4347-b3aa-6268aae6ceda-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cvz5g\" (UID: \"d52ce2cb-6893-4347-b3aa-6268aae6ceda\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:23.883590 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.883582 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52ce2cb-6893-4347-b3aa-6268aae6ceda-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cvz5g\" (UID: \"d52ce2cb-6893-4347-b3aa-6268aae6ceda\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:23.883896 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.883630 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52ce2cb-6893-4347-b3aa-6268aae6ceda-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cvz5g\" (UID: \"d52ce2cb-6893-4347-b3aa-6268aae6ceda\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:23.883896 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.883794 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52ce2cb-6893-4347-b3aa-6268aae6ceda-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cvz5g\" (UID: \"d52ce2cb-6893-4347-b3aa-6268aae6ceda\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:23.883896 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.883826 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52ce2cb-6893-4347-b3aa-6268aae6ceda-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cvz5g\" (UID: \"d52ce2cb-6893-4347-b3aa-6268aae6ceda\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:23.884055 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.883906 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52ce2cb-6893-4347-b3aa-6268aae6ceda-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cvz5g\" (UID: \"d52ce2cb-6893-4347-b3aa-6268aae6ceda\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:23.884055 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.883995 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52ce2cb-6893-4347-b3aa-6268aae6ceda-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cvz5g\" (UID: \"d52ce2cb-6893-4347-b3aa-6268aae6ceda\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:23.884490 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.884469 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52ce2cb-6893-4347-b3aa-6268aae6ceda-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cvz5g\" (UID: \"d52ce2cb-6893-4347-b3aa-6268aae6ceda\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:23.885757 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.885727 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52ce2cb-6893-4347-b3aa-6268aae6ceda-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cvz5g\" (UID: \"d52ce2cb-6893-4347-b3aa-6268aae6ceda\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:23.885981 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.885961 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52ce2cb-6893-4347-b3aa-6268aae6ceda-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cvz5g\" (UID: \"d52ce2cb-6893-4347-b3aa-6268aae6ceda\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:23.892142 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.892106 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:23.937358 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.937295 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52ce2cb-6893-4347-b3aa-6268aae6ceda-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cvz5g\" (UID: \"d52ce2cb-6893-4347-b3aa-6268aae6ceda\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:23.937476 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:23.937447 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnk9j\" (UniqueName: \"kubernetes.io/projected/d52ce2cb-6893-4347-b3aa-6268aae6ceda-kube-api-access-tnk9j\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cvz5g\" (UID: \"d52ce2cb-6893-4347-b3aa-6268aae6ceda\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:24.007978 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:24.007939 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:24.103645 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:24.103611 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:24.308269 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:24.308228 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:24.426171 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:24.426123 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g"] Nov 02 21:14:24.465207 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:14:24.465169 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52ce2cb_6893_4347_b3aa_6268aae6ceda.slice/crio-d552d0c00ad1a6e9c4bb33029787f0807d7b3b8c1e705a905eeecf49181a7ab2 WatchSource:0}: Error finding container d552d0c00ad1a6e9c4bb33029787f0807d7b3b8c1e705a905eeecf49181a7ab2: Status 404 returned error can't find the container with id d552d0c00ad1a6e9c4bb33029787f0807d7b3b8c1e705a905eeecf49181a7ab2 Nov 02 21:14:24.467323 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:24.467293 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:14:24.467394 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:24.467367 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:14:24.467430 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:24.467400 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:14:24.519168 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:24.519120 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:24.573077 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:24.572981 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" event={"ID":"d52ce2cb-6893-4347-b3aa-6268aae6ceda","Type":"ContainerStarted","Data":"1b6329230e558ea002d83e538257b004e1a639a38f43402c1edba2981b075391"} Nov 02 21:14:24.573077 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:24.573054 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" event={"ID":"d52ce2cb-6893-4347-b3aa-6268aae6ceda","Type":"ContainerStarted","Data":"d552d0c00ad1a6e9c4bb33029787f0807d7b3b8c1e705a905eeecf49181a7ab2"} Nov 02 21:14:24.687292 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:24.687236 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" podStartSLOduration=1.68721913 podStartE2EDuration="1.68721913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:14:24.685294686 +0000 UTC m=+357.124869407" watchObservedRunningTime="2026-11-02 21:14:24.68721913 +0000 UTC m=+357.126793851" Nov 02 21:14:25.008651 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:25.008570 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:25.013451 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:25.013424 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:25.578011 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:25.577978 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:25.579162 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:25.579116 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cvz5g" Nov 02 21:14:27.150201 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:27.150164 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:27.154356 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:27.154327 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-lpcf6"] Nov 02 21:14:27.213671 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:27.213639 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-lpcf6"] Nov 02 21:14:27.213848 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:27.213756 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-lpcf6" Nov 02 21:14:27.262932 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:27.262894 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 21:14:27.308928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:27.308896 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pb7w5\"" Nov 02 21:14:27.317042 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:27.317009 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm4mn\" (UniqueName: \"kubernetes.io/projected/99ffad07-be47-457b-9ac4-0f33fb5cdc40-kube-api-access-mm4mn\") pod \"limitador-limitador-74dfcfb894-lpcf6\" (UID: \"99ffad07-be47-457b-9ac4-0f33fb5cdc40\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-lpcf6" Nov 02 21:14:27.317225 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:27.317047 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/99ffad07-be47-457b-9ac4-0f33fb5cdc40-config-file\") pod \"limitador-limitador-74dfcfb894-lpcf6\" (UID: \"99ffad07-be47-457b-9ac4-0f33fb5cdc40\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-lpcf6" Nov 02 21:14:27.391486 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:27.391448 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:27.402731 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:27.402661 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-lpcf6"] Nov 02 21:14:27.418007 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:27.417982 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm4mn\" (UniqueName: \"kubernetes.io/projected/99ffad07-be47-457b-9ac4-0f33fb5cdc40-kube-api-access-mm4mn\") pod \"limitador-limitador-74dfcfb894-lpcf6\" (UID: \"99ffad07-be47-457b-9ac4-0f33fb5cdc40\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-lpcf6" Nov 02 21:14:27.418166 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:27.418013 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/99ffad07-be47-457b-9ac4-0f33fb5cdc40-config-file\") pod \"limitador-limitador-74dfcfb894-lpcf6\" (UID: \"99ffad07-be47-457b-9ac4-0f33fb5cdc40\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-lpcf6" Nov 02 21:14:27.418615 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:27.418596 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/99ffad07-be47-457b-9ac4-0f33fb5cdc40-config-file\") pod \"limitador-limitador-74dfcfb894-lpcf6\" (UID: \"99ffad07-be47-457b-9ac4-0f33fb5cdc40\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-lpcf6" Nov 02 21:14:27.471529 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:27.471496 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm4mn\" (UniqueName: \"kubernetes.io/projected/99ffad07-be47-457b-9ac4-0f33fb5cdc40-kube-api-access-mm4mn\") pod \"limitador-limitador-74dfcfb894-lpcf6\" (UID: \"99ffad07-be47-457b-9ac4-0f33fb5cdc40\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-lpcf6" Nov 02 21:14:27.541870 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:27.541834 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-lpcf6" Nov 02 21:14:27.591901 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:27.591867 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:27.912738 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:27.912633 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-lpcf6"] Nov 02 21:14:27.914812 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:14:27.914781 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ffad07_be47_457b_9ac4_0f33fb5cdc40.slice/crio-f1d1fa8fa2ff693f15e99a775960d1dae041378d72e4d6f853b42d13b49466dc WatchSource:0}: Error finding container f1d1fa8fa2ff693f15e99a775960d1dae041378d72e4d6f853b42d13b49466dc: Status 404 returned error can't find the container with id f1d1fa8fa2ff693f15e99a775960d1dae041378d72e4d6f853b42d13b49466dc Nov 02 21:14:28.468108 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:14:28.468070 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:14:28.513803 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:28.513766 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:28.596402 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:28.596364 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-lpcf6" event={"ID":"99ffad07-be47-457b-9ac4-0f33fb5cdc40","Type":"ContainerStarted","Data":"f1d1fa8fa2ff693f15e99a775960d1dae041378d72e4d6f853b42d13b49466dc"} Nov 02 21:14:28.721345 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:28.721245 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:30.437427 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:30.437393 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h8tn6"] Nov 02 21:14:30.440935 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:30.440916 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-h8tn6" Nov 02 21:14:30.490184 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:30.490151 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h8tn6"] Nov 02 21:14:30.534392 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:30.534364 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-tr6zv\"" Nov 02 21:14:30.545973 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:30.545949 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmstj\" (UniqueName: \"kubernetes.io/projected/142d92db-9ac9-4762-b65b-b01af806cdfa-kube-api-access-bmstj\") pod \"authorino-56c6c9ddd5-h8tn6\" (UID: \"142d92db-9ac9-4762-b65b-b01af806cdfa\") " pod="kuadrant-system/authorino-56c6c9ddd5-h8tn6" Nov 02 21:14:30.646897 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:30.646862 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmstj\" (UniqueName: \"kubernetes.io/projected/142d92db-9ac9-4762-b65b-b01af806cdfa-kube-api-access-bmstj\") pod \"authorino-56c6c9ddd5-h8tn6\" (UID: \"142d92db-9ac9-4762-b65b-b01af806cdfa\") " pod="kuadrant-system/authorino-56c6c9ddd5-h8tn6" Nov 02 21:14:30.700248 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:30.700179 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmstj\" (UniqueName: \"kubernetes.io/projected/142d92db-9ac9-4762-b65b-b01af806cdfa-kube-api-access-bmstj\") pod \"authorino-56c6c9ddd5-h8tn6\" (UID: \"142d92db-9ac9-4762-b65b-b01af806cdfa\") " pod="kuadrant-system/authorino-56c6c9ddd5-h8tn6" Nov 02 21:14:30.754998 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:30.754947 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:30.764091 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:30.763762 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-h8tn6" Nov 02 21:14:31.132383 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:31.132281 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h8tn6"] Nov 02 21:14:31.135206 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:14:31.135172 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod142d92db_9ac9_4762_b65b_b01af806cdfa.slice/crio-3d4241b9fc688cdffd76823fa0e282922d768978ed99ea1a35e339b99847ff76 WatchSource:0}: Error finding container 3d4241b9fc688cdffd76823fa0e282922d768978ed99ea1a35e339b99847ff76: Status 404 returned error can't find the container with id 3d4241b9fc688cdffd76823fa0e282922d768978ed99ea1a35e339b99847ff76 Nov 02 21:14:31.611304 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:31.611266 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-h8tn6" event={"ID":"142d92db-9ac9-4762-b65b-b01af806cdfa","Type":"ContainerStarted","Data":"3d4241b9fc688cdffd76823fa0e282922d768978ed99ea1a35e339b99847ff76"} Nov 02 21:14:31.612871 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:31.612840 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-lpcf6" event={"ID":"99ffad07-be47-457b-9ac4-0f33fb5cdc40","Type":"ContainerStarted","Data":"786e9533cde79c2083ac1ccff4f3322df8b11dba75da9c2208c571223042da07"} Nov 02 21:14:31.613004 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:31.612991 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-lpcf6" Nov 02 21:14:31.720744 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:31.720691 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-lpcf6" podStartSLOduration=1.593328784 podStartE2EDuration="4.720675808s" totalImagesPullingTime="3.127347024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:14:31.718214468 +0000 UTC m=+364.157789190" watchObservedRunningTime="2026-11-02 21:14:31.720675808 +0000 UTC m=+364.160250544" Nov 02 21:14:34.253594 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:34.253504 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:34.628964 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:34.628926 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-h8tn6" event={"ID":"142d92db-9ac9-4762-b65b-b01af806cdfa","Type":"ContainerStarted","Data":"49ffb36b46c8855b265bc9732fe1d45357a21dfce48bea6f8b03cde723555164"} Nov 02 21:14:34.734557 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:34.734489 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-h8tn6" podStartSLOduration=2.091644645 podStartE2EDuration="4.73447285s" totalImagesPullingTime="2.642828205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:14:34.733092232 +0000 UTC m=+367.172666953" watchObservedRunningTime="2026-11-02 21:14:34.73447285 +0000 UTC m=+367.174047572" Nov 02 21:14:34.783241 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:34.783210 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-dhqhh"] Nov 02 21:14:34.787115 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:34.787091 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dhqhh" Nov 02 21:14:34.836036 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:34.836003 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 21:14:34.836200 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:34.836058 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 21:14:34.881389 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:34.881306 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c6839ea0-0dae-40a5-a5c3-94220c3c70de-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-dhqhh\" (UID: \"c6839ea0-0dae-40a5-a5c3-94220c3c70de\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dhqhh" Nov 02 21:14:34.881551 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:34.881467 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6839ea0-0dae-40a5-a5c3-94220c3c70de-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-dhqhh\" (UID: \"c6839ea0-0dae-40a5-a5c3-94220c3c70de\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dhqhh" Nov 02 21:14:34.881619 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:34.881561 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6839ea0-0dae-40a5-a5c3-94220c3c70de-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-dhqhh\" (UID: \"c6839ea0-0dae-40a5-a5c3-94220c3c70de\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dhqhh" Nov 02 21:14:34.881691 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:34.881671 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49689\" (UniqueName: \"kubernetes.io/projected/c6839ea0-0dae-40a5-a5c3-94220c3c70de-kube-api-access-49689\") pod \"kserve-module-controller-manager-7566cf5fc8-dhqhh\" (UID: \"c6839ea0-0dae-40a5-a5c3-94220c3c70de\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dhqhh" Nov 02 21:14:34.882315 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:34.882297 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-jjglx\"" Nov 02 21:14:34.975636 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:34.975602 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 02 21:14:34.977644 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:34.977624 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-dhqhh"] Nov 02 21:14:34.982227 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:34.982200 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c6839ea0-0dae-40a5-a5c3-94220c3c70de-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-dhqhh\" (UID: \"c6839ea0-0dae-40a5-a5c3-94220c3c70de\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dhqhh" Nov 02 21:14:34.982349 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:34.982247 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6839ea0-0dae-40a5-a5c3-94220c3c70de-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-dhqhh\" (UID: \"c6839ea0-0dae-40a5-a5c3-94220c3c70de\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dhqhh" Nov 02 21:14:34.982349 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:34.982269 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6839ea0-0dae-40a5-a5c3-94220c3c70de-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-dhqhh\" (UID: \"c6839ea0-0dae-40a5-a5c3-94220c3c70de\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dhqhh" Nov 02 21:14:34.982349 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:34.982339 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49689\" (UniqueName: \"kubernetes.io/projected/c6839ea0-0dae-40a5-a5c3-94220c3c70de-kube-api-access-49689\") pod \"kserve-module-controller-manager-7566cf5fc8-dhqhh\" (UID: \"c6839ea0-0dae-40a5-a5c3-94220c3c70de\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dhqhh" Nov 02 21:14:34.982634 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:34.982603 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c6839ea0-0dae-40a5-a5c3-94220c3c70de-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-dhqhh\" (UID: \"c6839ea0-0dae-40a5-a5c3-94220c3c70de\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dhqhh" Nov 02 21:14:34.982716 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:34.982648 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6839ea0-0dae-40a5-a5c3-94220c3c70de-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-dhqhh\" (UID: \"c6839ea0-0dae-40a5-a5c3-94220c3c70de\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dhqhh" Nov 02 21:14:34.983122 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:34.983099 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6839ea0-0dae-40a5-a5c3-94220c3c70de-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-dhqhh\" (UID: \"c6839ea0-0dae-40a5-a5c3-94220c3c70de\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dhqhh" Nov 02 21:14:35.039977 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:35.039942 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49689\" (UniqueName: \"kubernetes.io/projected/c6839ea0-0dae-40a5-a5c3-94220c3c70de-kube-api-access-49689\") pod \"kserve-module-controller-manager-7566cf5fc8-dhqhh\" (UID: \"c6839ea0-0dae-40a5-a5c3-94220c3c70de\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dhqhh" Nov 02 21:14:35.100965 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:35.100932 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dhqhh" Nov 02 21:14:35.518588 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:35.518547 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-dhqhh"] Nov 02 21:14:35.521668 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:14:35.521633 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6839ea0_0dae_40a5_a5c3_94220c3c70de.slice/crio-dce17f8794b30bd98f3b32a6f1593595c62c7de6c6c95115fe33a639ebcd0fab WatchSource:0}: Error finding container dce17f8794b30bd98f3b32a6f1593595c62c7de6c6c95115fe33a639ebcd0fab: Status 404 returned error can't find the container with id dce17f8794b30bd98f3b32a6f1593595c62c7de6c6c95115fe33a639ebcd0fab Nov 02 21:14:35.635545 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:35.635505 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dhqhh" event={"ID":"c6839ea0-0dae-40a5-a5c3-94220c3c70de","Type":"ContainerStarted","Data":"dce17f8794b30bd98f3b32a6f1593595c62c7de6c6c95115fe33a639ebcd0fab"} Nov 02 21:14:39.658597 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:39.658553 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dhqhh" event={"ID":"c6839ea0-0dae-40a5-a5c3-94220c3c70de","Type":"ContainerStarted","Data":"35a6b5f621f47344544709726b36c7e53b5ea38ce730529bcfd9d83c99d80518"} Nov 02 21:14:39.659033 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:39.658696 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dhqhh" Nov 02 21:14:39.813375 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:39.813321 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dhqhh" podStartSLOduration=2.534770399 podStartE2EDuration="5.813305296s" totalImagesPullingTime="3.278534897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:14:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:14:39.81201433 +0000 UTC m=+372.251589052" watchObservedRunningTime="2026-11-02 21:14:39.813305296 +0000 UTC m=+372.252880058" Nov 02 21:14:42.619507 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:42.619481 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-lpcf6" Nov 02 21:14:45.363984 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:45.363950 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-lpcf6"] Nov 02 21:14:45.364456 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:45.364147 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-lpcf6" podUID="99ffad07-be47-457b-9ac4-0f33fb5cdc40" containerName="limitador" containerID="cri-o://786e9533cde79c2083ac1ccff4f3322df8b11dba75da9c2208c571223042da07" gracePeriod=30 Nov 02 21:14:45.906426 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:45.906396 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-lpcf6" Nov 02 21:14:45.981342 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:45.981265 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99ffad07-be47-457b-9ac4-0f33fb5cdc40-kube-api-access-mm4mn\" (UniqueName: \"kubernetes.io/projected/99ffad07-be47-457b-9ac4-0f33fb5cdc40-kube-api-access-mm4mn\") pod \"99ffad07-be47-457b-9ac4-0f33fb5cdc40\" (UID: \"99ffad07-be47-457b-9ac4-0f33fb5cdc40\") " Nov 02 21:14:45.981509 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:45.981341 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99ffad07-be47-457b-9ac4-0f33fb5cdc40-config-file\" (UniqueName: \"kubernetes.io/configmap/99ffad07-be47-457b-9ac4-0f33fb5cdc40-config-file\") pod \"99ffad07-be47-457b-9ac4-0f33fb5cdc40\" (UID: \"99ffad07-be47-457b-9ac4-0f33fb5cdc40\") " Nov 02 21:14:45.981751 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:45.981722 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99ffad07-be47-457b-9ac4-0f33fb5cdc40-config-file" pod "99ffad07-be47-457b-9ac4-0f33fb5cdc40" (UID: "99ffad07-be47-457b-9ac4-0f33fb5cdc40"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:14:45.983419 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:45.983397 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ffad07-be47-457b-9ac4-0f33fb5cdc40-kube-api-access-mm4mn" pod "99ffad07-be47-457b-9ac4-0f33fb5cdc40" (UID: "99ffad07-be47-457b-9ac4-0f33fb5cdc40"). InnerVolumeSpecName "kube-api-access-mm4mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:14:46.082645 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:46.082605 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm4mn\" (UniqueName: \"kubernetes.io/projected/99ffad07-be47-457b-9ac4-0f33fb5cdc40-kube-api-access-mm4mn\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:46.082645 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:46.082639 4935 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/99ffad07-be47-457b-9ac4-0f33fb5cdc40-config-file\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:46.697978 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:46.697937 4935 generic.go:356] "Generic (PLEG): container finished" podID="99ffad07-be47-457b-9ac4-0f33fb5cdc40" containerID="786e9533cde79c2083ac1ccff4f3322df8b11dba75da9c2208c571223042da07" exitCode=0 Nov 02 21:14:46.698501 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:46.697988 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-lpcf6" Nov 02 21:14:46.698501 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:46.698007 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-lpcf6" event={"ID":"99ffad07-be47-457b-9ac4-0f33fb5cdc40","Type":"ContainerDied","Data":"786e9533cde79c2083ac1ccff4f3322df8b11dba75da9c2208c571223042da07"} Nov 02 21:14:46.698501 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:46.698052 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-lpcf6" event={"ID":"99ffad07-be47-457b-9ac4-0f33fb5cdc40","Type":"ContainerDied","Data":"f1d1fa8fa2ff693f15e99a775960d1dae041378d72e4d6f853b42d13b49466dc"} Nov 02 21:14:46.698501 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:46.698075 4935 scope.go:122] "RemoveContainer" containerID="786e9533cde79c2083ac1ccff4f3322df8b11dba75da9c2208c571223042da07" Nov 02 21:14:46.707301 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:46.707279 4935 scope.go:122] "RemoveContainer" containerID="786e9533cde79c2083ac1ccff4f3322df8b11dba75da9c2208c571223042da07" Nov 02 21:14:46.707565 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:14:46.707548 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"786e9533cde79c2083ac1ccff4f3322df8b11dba75da9c2208c571223042da07\": container with ID starting with 786e9533cde79c2083ac1ccff4f3322df8b11dba75da9c2208c571223042da07 not found: ID does not exist" containerID="786e9533cde79c2083ac1ccff4f3322df8b11dba75da9c2208c571223042da07" Nov 02 21:14:46.707617 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:46.707575 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"786e9533cde79c2083ac1ccff4f3322df8b11dba75da9c2208c571223042da07"} err="failed to get container status \"786e9533cde79c2083ac1ccff4f3322df8b11dba75da9c2208c571223042da07\": rpc error: code = NotFound desc = could not find container \"786e9533cde79c2083ac1ccff4f3322df8b11dba75da9c2208c571223042da07\": container with ID starting with 786e9533cde79c2083ac1ccff4f3322df8b11dba75da9c2208c571223042da07 not found: ID does not exist" Nov 02 21:14:46.851095 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:46.851060 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-lpcf6"] Nov 02 21:14:46.897452 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:46.897422 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-lpcf6"] Nov 02 21:14:47.775077 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:47.775039 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99ffad07-be47-457b-9ac4-0f33fb5cdc40" path="/var/lib/kubelet/pods/99ffad07-be47-457b-9ac4-0f33fb5cdc40/volumes" Nov 02 21:14:53.649589 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:53.649551 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:53.893212 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:53.893175 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:54.212593 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:54.212557 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-z2tlv"] Nov 02 21:14:54.212944 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:54.212932 4935 cpu_manager.go:414] "containerMap: removing container" podUID="99ffad07-be47-457b-9ac4-0f33fb5cdc40" containerName="limitador" Nov 02 21:14:54.212987 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:54.212945 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99ffad07-be47-457b-9ac4-0f33fb5cdc40" containerName="limitador" Nov 02 21:14:54.213020 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:54.212998 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="99ffad07-be47-457b-9ac4-0f33fb5cdc40" containerName="limitador" Nov 02 21:14:54.217265 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:54.217244 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-z2tlv" Nov 02 21:14:54.258668 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:54.258633 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6pnq\" (UniqueName: \"kubernetes.io/projected/aa2ea710-8501-46f7-bb15-bca0cf39cb6f-kube-api-access-m6pnq\") pod \"postgres-7cdbf6dc44-z2tlv\" (UID: \"aa2ea710-8501-46f7-bb15-bca0cf39cb6f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-z2tlv" Nov 02 21:14:54.258816 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:54.258691 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/aa2ea710-8501-46f7-bb15-bca0cf39cb6f-data\") pod \"postgres-7cdbf6dc44-z2tlv\" (UID: \"aa2ea710-8501-46f7-bb15-bca0cf39cb6f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-z2tlv" Nov 02 21:14:54.268345 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:54.268308 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-z2tlv"] Nov 02 21:14:54.312237 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:54.312203 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:14:54.312237 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:54.312223 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:14:54.312237 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:54.312203 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 02 21:14:54.312458 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:54.312272 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-d9lln\"" Nov 02 21:14:54.359355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:54.359323 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/aa2ea710-8501-46f7-bb15-bca0cf39cb6f-data\") pod \"postgres-7cdbf6dc44-z2tlv\" (UID: \"aa2ea710-8501-46f7-bb15-bca0cf39cb6f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-z2tlv" Nov 02 21:14:54.359522 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:54.359409 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6pnq\" (UniqueName: \"kubernetes.io/projected/aa2ea710-8501-46f7-bb15-bca0cf39cb6f-kube-api-access-m6pnq\") pod \"postgres-7cdbf6dc44-z2tlv\" (UID: \"aa2ea710-8501-46f7-bb15-bca0cf39cb6f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-z2tlv" Nov 02 21:14:54.359731 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:54.359709 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/aa2ea710-8501-46f7-bb15-bca0cf39cb6f-data\") pod \"postgres-7cdbf6dc44-z2tlv\" (UID: \"aa2ea710-8501-46f7-bb15-bca0cf39cb6f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-z2tlv" Nov 02 21:14:54.375335 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:54.375303 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:14:54.412834 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:54.412803 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6pnq\" (UniqueName: \"kubernetes.io/projected/aa2ea710-8501-46f7-bb15-bca0cf39cb6f-kube-api-access-m6pnq\") pod \"postgres-7cdbf6dc44-z2tlv\" (UID: \"aa2ea710-8501-46f7-bb15-bca0cf39cb6f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-z2tlv" Nov 02 21:14:54.529768 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:54.529734 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-z2tlv" Nov 02 21:14:54.899875 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:54.899850 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-z2tlv"] Nov 02 21:14:54.901532 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:14:54.901504 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa2ea710_8501_46f7_bb15_bca0cf39cb6f.slice/crio-35099a62af821b72ea7157347d70bf0b517ccbbd719a208e21ac2b96ea8f8621 WatchSource:0}: Error finding container 35099a62af821b72ea7157347d70bf0b517ccbbd719a208e21ac2b96ea8f8621: Status 404 returned error can't find the container with id 35099a62af821b72ea7157347d70bf0b517ccbbd719a208e21ac2b96ea8f8621 Nov 02 21:14:55.748419 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:14:55.748379 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-z2tlv" event={"ID":"aa2ea710-8501-46f7-bb15-bca0cf39cb6f","Type":"ContainerStarted","Data":"35099a62af821b72ea7157347d70bf0b517ccbbd719a208e21ac2b96ea8f8621"} Nov 02 21:14:58.473406 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:14:58.473365 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:15:00.665036 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:00.665006 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dhqhh" Nov 02 21:15:03.793838 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:03.793802 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-z2tlv" event={"ID":"aa2ea710-8501-46f7-bb15-bca0cf39cb6f","Type":"ContainerStarted","Data":"1e7be7a6e90a51c67642ba0a70709e28daffe151b7e79c760db864acb80df969"} Nov 02 21:15:03.794278 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:03.793952 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-z2tlv" Nov 02 21:15:03.901064 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:03.901013 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-z2tlv" podStartSLOduration=1.496991274 podStartE2EDuration="9.900998718s" totalImagesPullingTime="8.404007444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:15:03.899889737 +0000 UTC m=+396.339464459" watchObservedRunningTime="2026-11-02 21:15:03.900998718 +0000 UTC m=+396.340573473" Nov 02 21:15:09.828766 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:09.828696 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-z2tlv" Nov 02 21:15:16.646641 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:16.646596 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-86bcf64849-h8nrb"] Nov 02 21:15:16.650437 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:16.650411 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86bcf64849-h8nrb" Nov 02 21:15:16.668865 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:16.668827 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:16.698730 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:16.698691 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 21:15:16.698911 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:16.698748 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 21:15:16.745121 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:16.745081 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mvsgp\"" Nov 02 21:15:16.745351 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:16.745307 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 21:15:16.747452 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:16.747426 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86bcf64849-h8nrb"] Nov 02 21:15:16.771798 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:16.771770 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kllnr\" (UniqueName: \"kubernetes.io/projected/f467a052-35e7-41dc-9093-ea9b8cb94442-kube-api-access-kllnr\") pod \"maas-controller-86bcf64849-h8nrb\" (UID: \"f467a052-35e7-41dc-9093-ea9b8cb94442\") " pod="opendatahub/maas-controller-86bcf64849-h8nrb" Nov 02 21:15:16.771953 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:16.771815 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f467a052-35e7-41dc-9093-ea9b8cb94442-metrics-certs\") pod \"maas-controller-86bcf64849-h8nrb\" (UID: \"f467a052-35e7-41dc-9093-ea9b8cb94442\") " pod="opendatahub/maas-controller-86bcf64849-h8nrb" Nov 02 21:15:16.771953 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:16.771912 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f467a052-35e7-41dc-9093-ea9b8cb94442-webhook-cert\") pod \"maas-controller-86bcf64849-h8nrb\" (UID: \"f467a052-35e7-41dc-9093-ea9b8cb94442\") " pod="opendatahub/maas-controller-86bcf64849-h8nrb" Nov 02 21:15:16.872925 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:16.872885 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kllnr\" (UniqueName: \"kubernetes.io/projected/f467a052-35e7-41dc-9093-ea9b8cb94442-kube-api-access-kllnr\") pod \"maas-controller-86bcf64849-h8nrb\" (UID: \"f467a052-35e7-41dc-9093-ea9b8cb94442\") " pod="opendatahub/maas-controller-86bcf64849-h8nrb" Nov 02 21:15:16.873177 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:16.872932 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f467a052-35e7-41dc-9093-ea9b8cb94442-metrics-certs\") pod \"maas-controller-86bcf64849-h8nrb\" (UID: \"f467a052-35e7-41dc-9093-ea9b8cb94442\") " pod="opendatahub/maas-controller-86bcf64849-h8nrb" Nov 02 21:15:16.873177 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:16.872998 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f467a052-35e7-41dc-9093-ea9b8cb94442-webhook-cert\") pod \"maas-controller-86bcf64849-h8nrb\" (UID: \"f467a052-35e7-41dc-9093-ea9b8cb94442\") " pod="opendatahub/maas-controller-86bcf64849-h8nrb" Nov 02 21:15:16.875618 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:16.875589 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f467a052-35e7-41dc-9093-ea9b8cb94442-metrics-certs\") pod \"maas-controller-86bcf64849-h8nrb\" (UID: \"f467a052-35e7-41dc-9093-ea9b8cb94442\") " pod="opendatahub/maas-controller-86bcf64849-h8nrb" Nov 02 21:15:16.875693 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:16.875670 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f467a052-35e7-41dc-9093-ea9b8cb94442-webhook-cert\") pod \"maas-controller-86bcf64849-h8nrb\" (UID: \"f467a052-35e7-41dc-9093-ea9b8cb94442\") " pod="opendatahub/maas-controller-86bcf64849-h8nrb" Nov 02 21:15:16.892813 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:16.892781 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:16.926651 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:16.926584 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kllnr\" (UniqueName: \"kubernetes.io/projected/f467a052-35e7-41dc-9093-ea9b8cb94442-kube-api-access-kllnr\") pod \"maas-controller-86bcf64849-h8nrb\" (UID: \"f467a052-35e7-41dc-9093-ea9b8cb94442\") " pod="opendatahub/maas-controller-86bcf64849-h8nrb" Nov 02 21:15:16.963282 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:16.963250 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86bcf64849-h8nrb" Nov 02 21:15:17.096113 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:17.096080 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:17.214069 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:17.213987 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d468bc7cc-hj64v"] Nov 02 21:15:17.217802 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:17.217780 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d468bc7cc-hj64v" Nov 02 21:15:17.271853 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:17.271818 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d468bc7cc-hj64v"] Nov 02 21:15:17.377140 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:17.377108 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7e1298eb-5e6b-4e00-9269-d88a130d72c7-webhook-cert\") pod \"maas-controller-7d468bc7cc-hj64v\" (UID: \"7e1298eb-5e6b-4e00-9269-d88a130d72c7\") " pod="opendatahub/maas-controller-7d468bc7cc-hj64v" Nov 02 21:15:17.377314 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:17.377239 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e1298eb-5e6b-4e00-9269-d88a130d72c7-metrics-certs\") pod \"maas-controller-7d468bc7cc-hj64v\" (UID: \"7e1298eb-5e6b-4e00-9269-d88a130d72c7\") " pod="opendatahub/maas-controller-7d468bc7cc-hj64v" Nov 02 21:15:17.377356 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:17.377314 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j7p7\" (UniqueName: \"kubernetes.io/projected/7e1298eb-5e6b-4e00-9269-d88a130d72c7-kube-api-access-6j7p7\") pod \"maas-controller-7d468bc7cc-hj64v\" (UID: \"7e1298eb-5e6b-4e00-9269-d88a130d72c7\") " pod="opendatahub/maas-controller-7d468bc7cc-hj64v" Nov 02 21:15:17.387845 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:17.387824 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86bcf64849-h8nrb"] Nov 02 21:15:17.389891 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:15:17.389870 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf467a052_35e7_41dc_9093_ea9b8cb94442.slice/crio-6f1ecfd70b5bba6d102ece1e620433b5448908e10ed16ddffc2dc3468582e1cc WatchSource:0}: Error finding container 6f1ecfd70b5bba6d102ece1e620433b5448908e10ed16ddffc2dc3468582e1cc: Status 404 returned error can't find the container with id 6f1ecfd70b5bba6d102ece1e620433b5448908e10ed16ddffc2dc3468582e1cc Nov 02 21:15:17.477960 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:17.477872 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7e1298eb-5e6b-4e00-9269-d88a130d72c7-webhook-cert\") pod \"maas-controller-7d468bc7cc-hj64v\" (UID: \"7e1298eb-5e6b-4e00-9269-d88a130d72c7\") " pod="opendatahub/maas-controller-7d468bc7cc-hj64v" Nov 02 21:15:17.477960 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:17.477940 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e1298eb-5e6b-4e00-9269-d88a130d72c7-metrics-certs\") pod \"maas-controller-7d468bc7cc-hj64v\" (UID: \"7e1298eb-5e6b-4e00-9269-d88a130d72c7\") " pod="opendatahub/maas-controller-7d468bc7cc-hj64v" Nov 02 21:15:17.478233 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:17.477983 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j7p7\" (UniqueName: \"kubernetes.io/projected/7e1298eb-5e6b-4e00-9269-d88a130d72c7-kube-api-access-6j7p7\") pod \"maas-controller-7d468bc7cc-hj64v\" (UID: \"7e1298eb-5e6b-4e00-9269-d88a130d72c7\") " pod="opendatahub/maas-controller-7d468bc7cc-hj64v" Nov 02 21:15:17.480547 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:17.480523 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e1298eb-5e6b-4e00-9269-d88a130d72c7-metrics-certs\") pod \"maas-controller-7d468bc7cc-hj64v\" (UID: \"7e1298eb-5e6b-4e00-9269-d88a130d72c7\") " pod="opendatahub/maas-controller-7d468bc7cc-hj64v" Nov 02 21:15:17.480640 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:17.480523 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7e1298eb-5e6b-4e00-9269-d88a130d72c7-webhook-cert\") pod \"maas-controller-7d468bc7cc-hj64v\" (UID: \"7e1298eb-5e6b-4e00-9269-d88a130d72c7\") " pod="opendatahub/maas-controller-7d468bc7cc-hj64v" Nov 02 21:15:17.531579 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:17.531543 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j7p7\" (UniqueName: \"kubernetes.io/projected/7e1298eb-5e6b-4e00-9269-d88a130d72c7-kube-api-access-6j7p7\") pod \"maas-controller-7d468bc7cc-hj64v\" (UID: \"7e1298eb-5e6b-4e00-9269-d88a130d72c7\") " pod="opendatahub/maas-controller-7d468bc7cc-hj64v" Nov 02 21:15:17.831592 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:17.831558 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d468bc7cc-hj64v" Nov 02 21:15:17.868847 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:17.868814 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86bcf64849-h8nrb" event={"ID":"f467a052-35e7-41dc-9093-ea9b8cb94442","Type":"ContainerStarted","Data":"6f1ecfd70b5bba6d102ece1e620433b5448908e10ed16ddffc2dc3468582e1cc"} Nov 02 21:15:18.197157 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:18.197048 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d468bc7cc-hj64v"] Nov 02 21:15:18.231681 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:15:18.231639 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e1298eb_5e6b_4e00_9269_d88a130d72c7.slice/crio-1981f9ba0661419c163b242827fcb7e0834eedee1e66c09c1e5438e813cf38ca WatchSource:0}: Error finding container 1981f9ba0661419c163b242827fcb7e0834eedee1e66c09c1e5438e813cf38ca: Status 404 returned error can't find the container with id 1981f9ba0661419c163b242827fcb7e0834eedee1e66c09c1e5438e813cf38ca Nov 02 21:15:18.874363 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:18.874324 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d468bc7cc-hj64v" event={"ID":"7e1298eb-5e6b-4e00-9269-d88a130d72c7","Type":"ContainerStarted","Data":"1981f9ba0661419c163b242827fcb7e0834eedee1e66c09c1e5438e813cf38ca"} Nov 02 21:15:19.625354 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:19.625314 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:24.843210 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:24.843174 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:24.909170 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:24.909114 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86bcf64849-h8nrb" event={"ID":"f467a052-35e7-41dc-9093-ea9b8cb94442","Type":"ContainerStarted","Data":"65ca9cfc11ba3e2184e305ac35d27005fe9cee1f87fd0498068d581f1d1de4ed"} Nov 02 21:15:24.909340 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:24.909199 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-86bcf64849-h8nrb" Nov 02 21:15:25.019726 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:25.019660 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-86bcf64849-h8nrb" podStartSLOduration=2.015309939 podStartE2EDuration="9.019640544s" totalImagesPullingTime="7.004330605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:15:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:15:25.018902661 +0000 UTC m=+417.458477386" watchObservedRunningTime="2026-11-02 21:15:25.019640544 +0000 UTC m=+417.459215278" Nov 02 21:15:25.054459 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:25.054415 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:25.916893 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:25.916843 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d468bc7cc-hj64v" event={"ID":"7e1298eb-5e6b-4e00-9269-d88a130d72c7","Type":"ContainerStarted","Data":"d04c62f90ac285517935481f026be68907815ec8e7c7f6c523bb29774be7ebc6"} Nov 02 21:15:25.917360 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:25.916948 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d468bc7cc-hj64v" Nov 02 21:15:26.025992 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:26.025934 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7d468bc7cc-hj64v" podStartSLOduration=1.9449337469999999 podStartE2EDuration="9.025918258s" totalImagesPullingTime="7.080984511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:15:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:15:26.024790473 +0000 UTC m=+418.464365196" watchObservedRunningTime="2026-11-02 21:15:26.025918258 +0000 UTC m=+418.465493013" Nov 02 21:15:27.965262 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:27.965229 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-nmwv9"] Nov 02 21:15:27.970046 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:27.970022 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-nmwv9" Nov 02 21:15:28.020299 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:28.020267 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 02 21:15:28.020299 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:28.020286 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 02 21:15:28.020299 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:28.020279 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 02 21:15:28.020573 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:28.020519 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-6nsfw\"" Nov 02 21:15:28.022710 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:28.022685 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-nmwv9"] Nov 02 21:15:28.077827 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:28.077726 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0d2b5b50-c80c-4bac-8830-6c38da5fee20-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-nmwv9\" (UID: \"0d2b5b50-c80c-4bac-8830-6c38da5fee20\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-nmwv9" Nov 02 21:15:28.077998 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:28.077868 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2b5b50-c80c-4bac-8830-6c38da5fee20-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-nmwv9\" (UID: \"0d2b5b50-c80c-4bac-8830-6c38da5fee20\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-nmwv9" Nov 02 21:15:28.077998 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:28.077951 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/0d2b5b50-c80c-4bac-8830-6c38da5fee20-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-nmwv9\" (UID: \"0d2b5b50-c80c-4bac-8830-6c38da5fee20\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-nmwv9" Nov 02 21:15:28.078122 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:28.078023 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxlgd\" (UniqueName: \"kubernetes.io/projected/0d2b5b50-c80c-4bac-8830-6c38da5fee20-kube-api-access-hxlgd\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-nmwv9\" (UID: \"0d2b5b50-c80c-4bac-8830-6c38da5fee20\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-nmwv9" Nov 02 21:15:28.179508 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:28.179470 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/0d2b5b50-c80c-4bac-8830-6c38da5fee20-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-nmwv9\" (UID: \"0d2b5b50-c80c-4bac-8830-6c38da5fee20\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-nmwv9" Nov 02 21:15:28.179697 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:28.179545 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxlgd\" (UniqueName: \"kubernetes.io/projected/0d2b5b50-c80c-4bac-8830-6c38da5fee20-kube-api-access-hxlgd\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-nmwv9\" (UID: \"0d2b5b50-c80c-4bac-8830-6c38da5fee20\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-nmwv9" Nov 02 21:15:28.179697 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:15:28.179601 4935 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 02 21:15:28.179697 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:15:28.179680 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d2b5b50-c80c-4bac-8830-6c38da5fee20-proxy-source podName:0d2b5b50-c80c-4bac-8830-6c38da5fee20 nodeName:}" failed. No retries permitted until 2026-11-02 21:15:28.679661643 +0000 UTC m=+421.119236361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/0d2b5b50-c80c-4bac-8830-6c38da5fee20-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-nmwv9" (UID: "0d2b5b50-c80c-4bac-8830-6c38da5fee20") : configmap "usage-tenancy-proxy-source" not found Nov 02 21:15:28.179867 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:28.179697 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0d2b5b50-c80c-4bac-8830-6c38da5fee20-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-nmwv9\" (UID: \"0d2b5b50-c80c-4bac-8830-6c38da5fee20\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-nmwv9" Nov 02 21:15:28.179867 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:28.179757 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2b5b50-c80c-4bac-8830-6c38da5fee20-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-nmwv9\" (UID: \"0d2b5b50-c80c-4bac-8830-6c38da5fee20\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-nmwv9" Nov 02 21:15:28.179967 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:15:28.179898 4935 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Nov 02 21:15:28.180020 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:15:28.179988 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d2b5b50-c80c-4bac-8830-6c38da5fee20-tls-cert podName:0d2b5b50-c80c-4bac-8830-6c38da5fee20 nodeName:}" failed. No retries permitted until 2026-11-02 21:15:28.679970569 +0000 UTC m=+421.119545302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/0d2b5b50-c80c-4bac-8830-6c38da5fee20-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-nmwv9" (UID: "0d2b5b50-c80c-4bac-8830-6c38da5fee20") : secret "usage-logs-tenancy-proxy-tls" not found Nov 02 21:15:28.180568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:28.180543 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2b5b50-c80c-4bac-8830-6c38da5fee20-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-nmwv9\" (UID: \"0d2b5b50-c80c-4bac-8830-6c38da5fee20\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-nmwv9" Nov 02 21:15:28.194829 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:28.194800 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:28.234266 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:28.234237 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxlgd\" (UniqueName: \"kubernetes.io/projected/0d2b5b50-c80c-4bac-8830-6c38da5fee20-kube-api-access-hxlgd\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-nmwv9\" (UID: \"0d2b5b50-c80c-4bac-8830-6c38da5fee20\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-nmwv9" Nov 02 21:15:28.468811 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:15:28.468729 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:15:28.473511 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:28.473485 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:28.683985 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:28.683951 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0d2b5b50-c80c-4bac-8830-6c38da5fee20-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-nmwv9\" (UID: \"0d2b5b50-c80c-4bac-8830-6c38da5fee20\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-nmwv9" Nov 02 21:15:28.684173 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:28.684094 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/0d2b5b50-c80c-4bac-8830-6c38da5fee20-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-nmwv9\" (UID: \"0d2b5b50-c80c-4bac-8830-6c38da5fee20\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-nmwv9" Nov 02 21:15:28.684849 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:28.684823 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/0d2b5b50-c80c-4bac-8830-6c38da5fee20-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-nmwv9\" (UID: \"0d2b5b50-c80c-4bac-8830-6c38da5fee20\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-nmwv9" Nov 02 21:15:28.686385 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:28.686362 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0d2b5b50-c80c-4bac-8830-6c38da5fee20-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-nmwv9\" (UID: \"0d2b5b50-c80c-4bac-8830-6c38da5fee20\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-nmwv9" Nov 02 21:15:28.885788 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:28.885754 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-nmwv9" Nov 02 21:15:29.262527 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:29.262493 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-nmwv9"] Nov 02 21:15:29.265239 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:15:29.265203 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2b5b50_c80c_4bac_8830_6c38da5fee20.slice/crio-8492ddceb242e26022fce9d7b3367d385677e5ba5ae66fb395d65150322fd567 WatchSource:0}: Error finding container 8492ddceb242e26022fce9d7b3367d385677e5ba5ae66fb395d65150322fd567: Status 404 returned error can't find the container with id 8492ddceb242e26022fce9d7b3367d385677e5ba5ae66fb395d65150322fd567 Nov 02 21:15:29.941633 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:29.941593 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-nmwv9" event={"ID":"0d2b5b50-c80c-4bac-8830-6c38da5fee20","Type":"ContainerStarted","Data":"8492ddceb242e26022fce9d7b3367d385677e5ba5ae66fb395d65150322fd567"} Nov 02 21:15:35.920159 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:35.920100 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-86bcf64849-h8nrb" Nov 02 21:15:36.924646 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:36.924614 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d468bc7cc-hj64v" Nov 02 21:15:37.082025 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:37.081943 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86bcf64849-h8nrb"] Nov 02 21:15:37.082243 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:37.082168 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-86bcf64849-h8nrb" podUID="f467a052-35e7-41dc-9093-ea9b8cb94442" containerName="manager" containerID="cri-o://65ca9cfc11ba3e2184e305ac35d27005fe9cee1f87fd0498068d581f1d1de4ed" gracePeriod=30 Nov 02 21:15:37.471566 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:37.471501 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86bcf64849-h8nrb" Nov 02 21:15:37.559355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:37.559323 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f467a052-35e7-41dc-9093-ea9b8cb94442-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f467a052-35e7-41dc-9093-ea9b8cb94442-metrics-certs\") pod \"f467a052-35e7-41dc-9093-ea9b8cb94442\" (UID: \"f467a052-35e7-41dc-9093-ea9b8cb94442\") " Nov 02 21:15:37.559555 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:37.559373 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f467a052-35e7-41dc-9093-ea9b8cb94442-webhook-cert\" (UniqueName: \"kubernetes.io/secret/f467a052-35e7-41dc-9093-ea9b8cb94442-webhook-cert\") pod \"f467a052-35e7-41dc-9093-ea9b8cb94442\" (UID: \"f467a052-35e7-41dc-9093-ea9b8cb94442\") " Nov 02 21:15:37.559555 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:37.559413 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f467a052-35e7-41dc-9093-ea9b8cb94442-kube-api-access-kllnr\" (UniqueName: \"kubernetes.io/projected/f467a052-35e7-41dc-9093-ea9b8cb94442-kube-api-access-kllnr\") pod \"f467a052-35e7-41dc-9093-ea9b8cb94442\" (UID: \"f467a052-35e7-41dc-9093-ea9b8cb94442\") " Nov 02 21:15:37.561623 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:37.561591 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f467a052-35e7-41dc-9093-ea9b8cb94442-webhook-cert" pod "f467a052-35e7-41dc-9093-ea9b8cb94442" (UID: "f467a052-35e7-41dc-9093-ea9b8cb94442"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:15:37.561623 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:37.561605 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f467a052-35e7-41dc-9093-ea9b8cb94442-metrics-certs" pod "f467a052-35e7-41dc-9093-ea9b8cb94442" (UID: "f467a052-35e7-41dc-9093-ea9b8cb94442"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:15:37.561771 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:37.561603 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f467a052-35e7-41dc-9093-ea9b8cb94442-kube-api-access-kllnr" pod "f467a052-35e7-41dc-9093-ea9b8cb94442" (UID: "f467a052-35e7-41dc-9093-ea9b8cb94442"). InnerVolumeSpecName "kube-api-access-kllnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:15:37.660307 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:37.660274 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f467a052-35e7-41dc-9093-ea9b8cb94442-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:37.660307 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:37.660301 4935 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f467a052-35e7-41dc-9093-ea9b8cb94442-webhook-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:37.660307 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:37.660315 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kllnr\" (UniqueName: \"kubernetes.io/projected/f467a052-35e7-41dc-9093-ea9b8cb94442-kube-api-access-kllnr\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:37.983008 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:37.982970 4935 generic.go:356] "Generic (PLEG): container finished" podID="f467a052-35e7-41dc-9093-ea9b8cb94442" containerID="65ca9cfc11ba3e2184e305ac35d27005fe9cee1f87fd0498068d581f1d1de4ed" exitCode=0 Nov 02 21:15:37.983501 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:37.983023 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86bcf64849-h8nrb" Nov 02 21:15:37.983501 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:37.983049 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86bcf64849-h8nrb" event={"ID":"f467a052-35e7-41dc-9093-ea9b8cb94442","Type":"ContainerDied","Data":"65ca9cfc11ba3e2184e305ac35d27005fe9cee1f87fd0498068d581f1d1de4ed"} Nov 02 21:15:37.983501 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:37.983120 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86bcf64849-h8nrb" event={"ID":"f467a052-35e7-41dc-9093-ea9b8cb94442","Type":"ContainerDied","Data":"6f1ecfd70b5bba6d102ece1e620433b5448908e10ed16ddffc2dc3468582e1cc"} Nov 02 21:15:37.983501 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:37.983164 4935 scope.go:122] "RemoveContainer" containerID="65ca9cfc11ba3e2184e305ac35d27005fe9cee1f87fd0498068d581f1d1de4ed" Nov 02 21:15:37.991936 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:37.991917 4935 scope.go:122] "RemoveContainer" containerID="65ca9cfc11ba3e2184e305ac35d27005fe9cee1f87fd0498068d581f1d1de4ed" Nov 02 21:15:37.992206 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:15:37.992184 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ca9cfc11ba3e2184e305ac35d27005fe9cee1f87fd0498068d581f1d1de4ed\": container with ID starting with 65ca9cfc11ba3e2184e305ac35d27005fe9cee1f87fd0498068d581f1d1de4ed not found: ID does not exist" containerID="65ca9cfc11ba3e2184e305ac35d27005fe9cee1f87fd0498068d581f1d1de4ed" Nov 02 21:15:37.992300 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:37.992213 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ca9cfc11ba3e2184e305ac35d27005fe9cee1f87fd0498068d581f1d1de4ed"} err="failed to get container status \"65ca9cfc11ba3e2184e305ac35d27005fe9cee1f87fd0498068d581f1d1de4ed\": rpc error: code = NotFound desc = could not find container \"65ca9cfc11ba3e2184e305ac35d27005fe9cee1f87fd0498068d581f1d1de4ed\": container with ID starting with 65ca9cfc11ba3e2184e305ac35d27005fe9cee1f87fd0498068d581f1d1de4ed not found: ID does not exist" Nov 02 21:15:38.135644 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:38.135557 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86bcf64849-h8nrb"] Nov 02 21:15:38.138681 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:38.138654 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-86bcf64849-h8nrb"] Nov 02 21:15:39.777387 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:39.777343 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f467a052-35e7-41dc-9093-ea9b8cb94442" path="/var/lib/kubelet/pods/f467a052-35e7-41dc-9093-ea9b8cb94442/volumes" Nov 02 21:15:42.065058 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:42.065020 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:42.292448 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:42.292418 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:42.514761 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:42.514721 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:42.758304 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:42.758272 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:42.979865 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:42.979796 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:43.002217 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:43.002180 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xbz5g"] Nov 02 21:15:43.002675 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:43.002654 4935 cpu_manager.go:414] "containerMap: removing container" podUID="f467a052-35e7-41dc-9093-ea9b8cb94442" containerName="manager" Nov 02 21:15:43.002764 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:43.002675 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f467a052-35e7-41dc-9093-ea9b8cb94442" containerName="manager" Nov 02 21:15:43.002823 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:43.002769 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="f467a052-35e7-41dc-9093-ea9b8cb94442" containerName="manager" Nov 02 21:15:43.007717 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:43.007695 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbz5g" Nov 02 21:15:43.059322 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:43.059296 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xbz5g"] Nov 02 21:15:43.102687 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:43.102633 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-88tjl\"" Nov 02 21:15:43.103121 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:43.102716 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 02 21:15:43.113687 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:43.113424 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-xbz5g\" (UID: \"87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbz5g" Nov 02 21:15:43.113687 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:43.113476 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgtfg\" (UniqueName: \"kubernetes.io/projected/87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4-kube-api-access-vgtfg\") pod \"llmisvc-controller-manager-869d5dcc5d-xbz5g\" (UID: \"87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbz5g" Nov 02 21:15:43.214581 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:43.214536 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-xbz5g\" (UID: \"87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbz5g" Nov 02 21:15:43.214778 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:43.214600 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgtfg\" (UniqueName: \"kubernetes.io/projected/87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4-kube-api-access-vgtfg\") pod \"llmisvc-controller-manager-869d5dcc5d-xbz5g\" (UID: \"87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbz5g" Nov 02 21:15:43.217544 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:43.217514 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-xbz5g\" (UID: \"87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbz5g" Nov 02 21:15:43.233730 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:43.233652 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:43.268203 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:43.268169 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgtfg\" (UniqueName: \"kubernetes.io/projected/87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4-kube-api-access-vgtfg\") pod \"llmisvc-controller-manager-869d5dcc5d-xbz5g\" (UID: \"87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbz5g" Nov 02 21:15:43.321011 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:43.320975 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbz5g" Nov 02 21:15:43.704994 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:43.704967 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xbz5g"] Nov 02 21:15:43.707391 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:15:43.707359 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod87f4d61d_3160_4910_b0c4_5f4b6eb5a6b4.slice/crio-66de1ce0a0af90e57f13459e307dd59b7cd84e11eac6e24b1789d35e681727ba WatchSource:0}: Error finding container 66de1ce0a0af90e57f13459e307dd59b7cd84e11eac6e24b1789d35e681727ba: Status 404 returned error can't find the container with id 66de1ce0a0af90e57f13459e307dd59b7cd84e11eac6e24b1789d35e681727ba Nov 02 21:15:44.024563 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:44.024524 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbz5g" event={"ID":"87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4","Type":"ContainerStarted","Data":"66de1ce0a0af90e57f13459e307dd59b7cd84e11eac6e24b1789d35e681727ba"} Nov 02 21:15:44.378869 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:44.378793 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:48.052971 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:48.052930 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbz5g" event={"ID":"87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4","Type":"ContainerStarted","Data":"88bcda2071216888d4bb995de86c97e9016d3bc8d26a4894bcbae6c6aacef975"} Nov 02 21:15:48.053526 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:48.053042 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbz5g" Nov 02 21:15:48.162187 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:48.162103 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbz5g" podStartSLOduration=2.890232001 podStartE2EDuration="6.162088042s" totalImagesPullingTime="3.271856041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:15:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:15:48.160797316 +0000 UTC m=+440.600372039" watchObservedRunningTime="2026-11-02 21:15:48.162088042 +0000 UTC m=+440.601662802" Nov 02 21:15:50.536980 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:50.536941 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:50.739984 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:50.739934 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:50.942122 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:50.942041 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:51.161296 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:51.161265 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:51.383120 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:51.383088 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:51.634769 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:51.634687 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:51.861532 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:51.861454 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:55.728784 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:55.728747 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:55.963108 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:55.963074 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:57.750821 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:57.750790 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:57.950303 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:57.950273 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:58.146670 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:58.146636 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:58.468055 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:15:58.467973 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:15:59.644547 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:59.644512 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:15:59.861804 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:15:59.861772 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:16:13.072450 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:13.072408 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:16:13.202575 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:13.202489 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-nmwv9" event={"ID":"0d2b5b50-c80c-4bac-8830-6c38da5fee20","Type":"ContainerStarted","Data":"61c795bb0b864de01c4477f746119490cf2cb761732ccb06ebc044b294b12223"} Nov 02 21:16:13.202575 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:13.202555 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-nmwv9" Nov 02 21:16:13.299023 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:13.298990 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:16:13.308941 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:13.308881 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-nmwv9" podStartSLOduration=2.662113298 podStartE2EDuration="46.308861029s" totalImagesPullingTime="43.646747731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:15:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:16:13.307753812 +0000 UTC m=+465.747328534" watchObservedRunningTime="2026-11-02 21:16:13.308861029 +0000 UTC m=+465.748435754" Nov 02 21:16:13.439388 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:13.439353 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-bbf944969-v2c9t"] Nov 02 21:16:13.442114 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:13.442090 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bbf944969-v2c9t" Nov 02 21:16:13.491177 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:13.491093 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 21:16:13.491177 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:13.491110 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 21:16:13.491177 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:13.491097 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-87fjb\"" Nov 02 21:16:13.499573 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:13.499546 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bbf944969-v2c9t"] Nov 02 21:16:13.525553 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:13.525522 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e97812f0-4d82-49eb-94b1-27274f2c0f90-openshift-service-ca\") pod \"maas-api-bbf944969-v2c9t\" (UID: \"e97812f0-4d82-49eb-94b1-27274f2c0f90\") " pod="odh-ai-gateway-infra/maas-api-bbf944969-v2c9t" Nov 02 21:16:13.525709 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:13.525572 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmsmn\" (UniqueName: \"kubernetes.io/projected/e97812f0-4d82-49eb-94b1-27274f2c0f90-kube-api-access-fmsmn\") pod \"maas-api-bbf944969-v2c9t\" (UID: \"e97812f0-4d82-49eb-94b1-27274f2c0f90\") " pod="odh-ai-gateway-infra/maas-api-bbf944969-v2c9t" Nov 02 21:16:13.525709 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:13.525668 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e97812f0-4d82-49eb-94b1-27274f2c0f90-maas-api-tls\") pod \"maas-api-bbf944969-v2c9t\" (UID: \"e97812f0-4d82-49eb-94b1-27274f2c0f90\") " pod="odh-ai-gateway-infra/maas-api-bbf944969-v2c9t" Nov 02 21:16:13.525709 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:13.525702 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e97812f0-4d82-49eb-94b1-27274f2c0f90-metrics-certs\") pod \"maas-api-bbf944969-v2c9t\" (UID: \"e97812f0-4d82-49eb-94b1-27274f2c0f90\") " pod="odh-ai-gateway-infra/maas-api-bbf944969-v2c9t" Nov 02 21:16:13.626775 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:13.626721 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e97812f0-4d82-49eb-94b1-27274f2c0f90-metrics-certs\") pod \"maas-api-bbf944969-v2c9t\" (UID: \"e97812f0-4d82-49eb-94b1-27274f2c0f90\") " pod="odh-ai-gateway-infra/maas-api-bbf944969-v2c9t" Nov 02 21:16:13.626960 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:13.626812 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e97812f0-4d82-49eb-94b1-27274f2c0f90-openshift-service-ca\") pod \"maas-api-bbf944969-v2c9t\" (UID: \"e97812f0-4d82-49eb-94b1-27274f2c0f90\") " pod="odh-ai-gateway-infra/maas-api-bbf944969-v2c9t" Nov 02 21:16:13.626960 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:13.626852 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmsmn\" (UniqueName: \"kubernetes.io/projected/e97812f0-4d82-49eb-94b1-27274f2c0f90-kube-api-access-fmsmn\") pod \"maas-api-bbf944969-v2c9t\" (UID: \"e97812f0-4d82-49eb-94b1-27274f2c0f90\") " pod="odh-ai-gateway-infra/maas-api-bbf944969-v2c9t" Nov 02 21:16:13.626960 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:13.626895 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e97812f0-4d82-49eb-94b1-27274f2c0f90-maas-api-tls\") pod \"maas-api-bbf944969-v2c9t\" (UID: \"e97812f0-4d82-49eb-94b1-27274f2c0f90\") " pod="odh-ai-gateway-infra/maas-api-bbf944969-v2c9t" Nov 02 21:16:13.627601 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:13.627576 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e97812f0-4d82-49eb-94b1-27274f2c0f90-openshift-service-ca\") pod \"maas-api-bbf944969-v2c9t\" (UID: \"e97812f0-4d82-49eb-94b1-27274f2c0f90\") " pod="odh-ai-gateway-infra/maas-api-bbf944969-v2c9t" Nov 02 21:16:13.629259 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:13.629236 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e97812f0-4d82-49eb-94b1-27274f2c0f90-metrics-certs\") pod \"maas-api-bbf944969-v2c9t\" (UID: \"e97812f0-4d82-49eb-94b1-27274f2c0f90\") " pod="odh-ai-gateway-infra/maas-api-bbf944969-v2c9t" Nov 02 21:16:13.629362 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:13.629319 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e97812f0-4d82-49eb-94b1-27274f2c0f90-maas-api-tls\") pod \"maas-api-bbf944969-v2c9t\" (UID: \"e97812f0-4d82-49eb-94b1-27274f2c0f90\") " pod="odh-ai-gateway-infra/maas-api-bbf944969-v2c9t" Nov 02 21:16:13.680962 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:13.680929 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmsmn\" (UniqueName: \"kubernetes.io/projected/e97812f0-4d82-49eb-94b1-27274f2c0f90-kube-api-access-fmsmn\") pod \"maas-api-bbf944969-v2c9t\" (UID: \"e97812f0-4d82-49eb-94b1-27274f2c0f90\") " pod="odh-ai-gateway-infra/maas-api-bbf944969-v2c9t" Nov 02 21:16:13.758398 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:13.758316 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bbf944969-v2c9t" Nov 02 21:16:14.026180 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:14.026145 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b548fd7c-bxrbs"] Nov 02 21:16:14.029699 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:14.029672 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b548fd7c-bxrbs" Nov 02 21:16:14.078766 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:14.078734 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 02 21:16:14.081023 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:14.080988 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b548fd7c-bxrbs"] Nov 02 21:16:14.133234 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:14.133193 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9ae05787-1ce2-40c7-ad94-e18443194248-tls-cert\") pod \"authorino-5b548fd7c-bxrbs\" (UID: \"9ae05787-1ce2-40c7-ad94-e18443194248\") " pod="kuadrant-system/authorino-5b548fd7c-bxrbs" Nov 02 21:16:14.133433 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:14.133267 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqx7d\" (UniqueName: \"kubernetes.io/projected/9ae05787-1ce2-40c7-ad94-e18443194248-kube-api-access-lqx7d\") pod \"authorino-5b548fd7c-bxrbs\" (UID: \"9ae05787-1ce2-40c7-ad94-e18443194248\") " pod="kuadrant-system/authorino-5b548fd7c-bxrbs" Nov 02 21:16:14.176654 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:14.176628 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bbf944969-v2c9t"] Nov 02 21:16:14.178378 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:16:14.178345 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97812f0_4d82_49eb_94b1_27274f2c0f90.slice/crio-d427c3a0add04d2cef27877966dc7a8ba2db60ecf587e10b9a31a06cf1f02395 WatchSource:0}: Error finding container d427c3a0add04d2cef27877966dc7a8ba2db60ecf587e10b9a31a06cf1f02395: Status 404 returned error can't find the container with id d427c3a0add04d2cef27877966dc7a8ba2db60ecf587e10b9a31a06cf1f02395 Nov 02 21:16:14.209409 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:14.209380 4935 generic.go:356] "Generic (PLEG): container finished" podID="d9f99d96-f28e-4152-8285-c6aefa58a412" containerID="8ec2bc86e31614b7af498248e22d49ba25a70a0e41d1293906ecefddc80fa3b9" exitCode=0 Nov 02 21:16:14.209524 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:14.209451 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" event={"ID":"d9f99d96-f28e-4152-8285-c6aefa58a412","Type":"ContainerDied","Data":"8ec2bc86e31614b7af498248e22d49ba25a70a0e41d1293906ecefddc80fa3b9"} Nov 02 21:16:14.209882 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:14.209860 4935 scope.go:122] "RemoveContainer" containerID="8ec2bc86e31614b7af498248e22d49ba25a70a0e41d1293906ecefddc80fa3b9" Nov 02 21:16:14.210825 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:14.210804 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bbf944969-v2c9t" event={"ID":"e97812f0-4d82-49eb-94b1-27274f2c0f90","Type":"ContainerStarted","Data":"d427c3a0add04d2cef27877966dc7a8ba2db60ecf587e10b9a31a06cf1f02395"} Nov 02 21:16:14.210982 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:14.210953 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:16:14.211028 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:14.211019 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:16:14.211071 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:14.211050 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:16:14.234453 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:14.234417 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9ae05787-1ce2-40c7-ad94-e18443194248-tls-cert\") pod \"authorino-5b548fd7c-bxrbs\" (UID: \"9ae05787-1ce2-40c7-ad94-e18443194248\") " pod="kuadrant-system/authorino-5b548fd7c-bxrbs" Nov 02 21:16:14.234579 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:14.234520 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqx7d\" (UniqueName: \"kubernetes.io/projected/9ae05787-1ce2-40c7-ad94-e18443194248-kube-api-access-lqx7d\") pod \"authorino-5b548fd7c-bxrbs\" (UID: \"9ae05787-1ce2-40c7-ad94-e18443194248\") " pod="kuadrant-system/authorino-5b548fd7c-bxrbs" Nov 02 21:16:14.236716 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:14.236697 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9ae05787-1ce2-40c7-ad94-e18443194248-tls-cert\") pod \"authorino-5b548fd7c-bxrbs\" (UID: \"9ae05787-1ce2-40c7-ad94-e18443194248\") " pod="kuadrant-system/authorino-5b548fd7c-bxrbs" Nov 02 21:16:14.287265 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:14.287210 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqx7d\" (UniqueName: \"kubernetes.io/projected/9ae05787-1ce2-40c7-ad94-e18443194248-kube-api-access-lqx7d\") pod \"authorino-5b548fd7c-bxrbs\" (UID: \"9ae05787-1ce2-40c7-ad94-e18443194248\") " pod="kuadrant-system/authorino-5b548fd7c-bxrbs" Nov 02 21:16:14.341573 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:14.341540 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b548fd7c-bxrbs" Nov 02 21:16:14.353078 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:14.353036 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:16:14.712532 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:14.712508 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b548fd7c-bxrbs"] Nov 02 21:16:14.715161 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:16:14.715120 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ae05787_1ce2_40c7_ad94_e18443194248.slice/crio-a2baec864efb778ed2790afd146b99ef0bf8efdc13ed79647c60c9868057f5b3 WatchSource:0}: Error finding container a2baec864efb778ed2790afd146b99ef0bf8efdc13ed79647c60c9868057f5b3: Status 404 returned error can't find the container with id a2baec864efb778ed2790afd146b99ef0bf8efdc13ed79647c60c9868057f5b3 Nov 02 21:16:15.222895 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:15.222852 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" event={"ID":"d9f99d96-f28e-4152-8285-c6aefa58a412","Type":"ContainerStarted","Data":"999eadc2a2bd21f645e23310962359cbc0ce580d8b2c780d4c8ad57052219b39"} Nov 02 21:16:15.250222 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:15.250177 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b548fd7c-bxrbs" event={"ID":"9ae05787-1ce2-40c7-ad94-e18443194248","Type":"ContainerStarted","Data":"a2baec864efb778ed2790afd146b99ef0bf8efdc13ed79647c60c9868057f5b3"} Nov 02 21:16:15.582523 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:15.582483 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:16:15.582741 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:15.582687 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4r6s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Nov 02 21:16:15.582816 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:15.582762 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" podUID="d9f99d96-f28e-4152-8285-c6aefa58a412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Nov 02 21:16:16.259701 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:16.259666 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b548fd7c-bxrbs" event={"ID":"9ae05787-1ce2-40c7-ad94-e18443194248","Type":"ContainerStarted","Data":"29a17ef97678f034737c8839cc92611aa0e8d3c1d212d1641135eba266ca0bc7"} Nov 02 21:16:16.369261 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:16.369090 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5b548fd7c-bxrbs" podStartSLOduration=1.971267965 podStartE2EDuration="3.369073446s" totalImagesPullingTime="1.397805481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:16:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:16:16.368058235 +0000 UTC m=+468.807632959" watchObservedRunningTime="2026-11-02 21:16:16.369073446 +0000 UTC m=+468.808648164" Nov 02 21:16:16.394346 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:16.394311 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h8tn6"] Nov 02 21:16:16.394843 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:16.394792 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-h8tn6" podUID="142d92db-9ac9-4762-b65b-b01af806cdfa" containerName="authorino" containerID="cri-o://49ffb36b46c8855b265bc9732fe1d45357a21dfce48bea6f8b03cde723555164" gracePeriod=30 Nov 02 21:16:16.582338 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:16.582249 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4r6s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Nov 02 21:16:16.582338 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:16.582294 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" podUID="d9f99d96-f28e-4152-8285-c6aefa58a412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Nov 02 21:16:16.693478 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:16.693448 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-h8tn6" Nov 02 21:16:16.770823 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:16.770787 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/142d92db-9ac9-4762-b65b-b01af806cdfa-kube-api-access-bmstj\" (UniqueName: \"kubernetes.io/projected/142d92db-9ac9-4762-b65b-b01af806cdfa-kube-api-access-bmstj\") pod \"142d92db-9ac9-4762-b65b-b01af806cdfa\" (UID: \"142d92db-9ac9-4762-b65b-b01af806cdfa\") " Nov 02 21:16:16.772815 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:16.772790 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/142d92db-9ac9-4762-b65b-b01af806cdfa-kube-api-access-bmstj" pod "142d92db-9ac9-4762-b65b-b01af806cdfa" (UID: "142d92db-9ac9-4762-b65b-b01af806cdfa"). InnerVolumeSpecName "kube-api-access-bmstj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:16:16.872285 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:16.872245 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmstj\" (UniqueName: \"kubernetes.io/projected/142d92db-9ac9-4762-b65b-b01af806cdfa-kube-api-access-bmstj\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:16:17.266924 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:17.266885 4935 generic.go:356] "Generic (PLEG): container finished" podID="142d92db-9ac9-4762-b65b-b01af806cdfa" containerID="49ffb36b46c8855b265bc9732fe1d45357a21dfce48bea6f8b03cde723555164" exitCode=0 Nov 02 21:16:17.267441 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:17.266992 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-h8tn6" event={"ID":"142d92db-9ac9-4762-b65b-b01af806cdfa","Type":"ContainerDied","Data":"49ffb36b46c8855b265bc9732fe1d45357a21dfce48bea6f8b03cde723555164"} Nov 02 21:16:17.267441 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:17.267032 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-h8tn6" event={"ID":"142d92db-9ac9-4762-b65b-b01af806cdfa","Type":"ContainerDied","Data":"3d4241b9fc688cdffd76823fa0e282922d768978ed99ea1a35e339b99847ff76"} Nov 02 21:16:17.267441 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:17.267054 4935 scope.go:122] "RemoveContainer" containerID="49ffb36b46c8855b265bc9732fe1d45357a21dfce48bea6f8b03cde723555164" Nov 02 21:16:17.267441 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:17.267255 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-h8tn6" Nov 02 21:16:17.270951 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:17.270917 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bbf944969-v2c9t" event={"ID":"e97812f0-4d82-49eb-94b1-27274f2c0f90","Type":"ContainerStarted","Data":"e5208b9d6fa7bc161be9cdff608571eeb1fab4077c1fece3872bf6ff1d4719f4"} Nov 02 21:16:17.271109 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:17.270964 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-bbf944969-v2c9t" Nov 02 21:16:17.278749 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:17.278725 4935 scope.go:122] "RemoveContainer" containerID="49ffb36b46c8855b265bc9732fe1d45357a21dfce48bea6f8b03cde723555164" Nov 02 21:16:17.279088 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:16:17.279069 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49ffb36b46c8855b265bc9732fe1d45357a21dfce48bea6f8b03cde723555164\": container with ID starting with 49ffb36b46c8855b265bc9732fe1d45357a21dfce48bea6f8b03cde723555164 not found: ID does not exist" containerID="49ffb36b46c8855b265bc9732fe1d45357a21dfce48bea6f8b03cde723555164" Nov 02 21:16:17.279164 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:17.279095 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ffb36b46c8855b265bc9732fe1d45357a21dfce48bea6f8b03cde723555164"} err="failed to get container status \"49ffb36b46c8855b265bc9732fe1d45357a21dfce48bea6f8b03cde723555164\": rpc error: code = NotFound desc = could not find container \"49ffb36b46c8855b265bc9732fe1d45357a21dfce48bea6f8b03cde723555164\": container with ID starting with 49ffb36b46c8855b265bc9732fe1d45357a21dfce48bea6f8b03cde723555164 not found: ID does not exist" Nov 02 21:16:17.437210 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:17.433186 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h8tn6"] Nov 02 21:16:17.437210 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:17.434703 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h8tn6"] Nov 02 21:16:17.571277 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:17.571214 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-bbf944969-v2c9t" podStartSLOduration=2.353527001 podStartE2EDuration="4.571198144s" totalImagesPullingTime="2.217671143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:16:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:16:17.570117015 +0000 UTC m=+470.009691736" watchObservedRunningTime="2026-11-02 21:16:17.571198144 +0000 UTC m=+470.010772866" Nov 02 21:16:17.583285 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:17.583250 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4r6s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Nov 02 21:16:17.583412 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:17.583303 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" podUID="d9f99d96-f28e-4152-8285-c6aefa58a412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Nov 02 21:16:17.775149 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:17.775106 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="142d92db-9ac9-4762-b65b-b01af806cdfa" path="/var/lib/kubelet/pods/142d92db-9ac9-4762-b65b-b01af806cdfa/volumes" Nov 02 21:16:18.582378 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:18.582341 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4r6s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Nov 02 21:16:18.582784 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:18.582393 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" podUID="d9f99d96-f28e-4152-8285-c6aefa58a412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Nov 02 21:16:19.059889 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:19.059836 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbz5g" Nov 02 21:16:19.583308 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:19.583273 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4r6s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Nov 02 21:16:19.583730 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:19.583323 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" podUID="d9f99d96-f28e-4152-8285-c6aefa58a412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Nov 02 21:16:19.996303 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:19.996216 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:16:20.586314 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:20.586282 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:16:20.586774 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:20.586570 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:16:20.587202 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:20.587184 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" Nov 02 21:16:23.282542 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:23.282507 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-bbf944969-v2c9t" Nov 02 21:16:24.217494 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:24.217463 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-nmwv9" Nov 02 21:16:28.468422 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:16:28.468382 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:16:39.675567 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:39.675478 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:16:39.900777 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:39.900745 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:16:57.614736 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:57.614693 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-28jss"] Nov 02 21:16:57.615258 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:57.615099 4935 cpu_manager.go:414] "containerMap: removing container" podUID="142d92db-9ac9-4762-b65b-b01af806cdfa" containerName="authorino" Nov 02 21:16:57.615258 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:57.615111 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="142d92db-9ac9-4762-b65b-b01af806cdfa" containerName="authorino" Nov 02 21:16:57.615258 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:57.615202 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="142d92db-9ac9-4762-b65b-b01af806cdfa" containerName="authorino" Nov 02 21:16:57.642487 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:57.642456 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-28jss" Nov 02 21:16:57.672488 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:57.672461 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-28jss"] Nov 02 21:16:57.738042 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:57.738005 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd2913d9-e59b-4b4a-b10a-a324e0c62f25-cert\") pod \"llmisvc-controller-manager-6466c554d8-28jss\" (UID: \"cd2913d9-e59b-4b4a-b10a-a324e0c62f25\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-28jss" Nov 02 21:16:57.738210 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:57.738051 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q782l\" (UniqueName: \"kubernetes.io/projected/cd2913d9-e59b-4b4a-b10a-a324e0c62f25-kube-api-access-q782l\") pod \"llmisvc-controller-manager-6466c554d8-28jss\" (UID: \"cd2913d9-e59b-4b4a-b10a-a324e0c62f25\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-28jss" Nov 02 21:16:57.839145 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:57.839094 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd2913d9-e59b-4b4a-b10a-a324e0c62f25-cert\") pod \"llmisvc-controller-manager-6466c554d8-28jss\" (UID: \"cd2913d9-e59b-4b4a-b10a-a324e0c62f25\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-28jss" Nov 02 21:16:57.839386 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:57.839180 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q782l\" (UniqueName: \"kubernetes.io/projected/cd2913d9-e59b-4b4a-b10a-a324e0c62f25-kube-api-access-q782l\") pod \"llmisvc-controller-manager-6466c554d8-28jss\" (UID: \"cd2913d9-e59b-4b4a-b10a-a324e0c62f25\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-28jss" Nov 02 21:16:57.841715 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:57.841679 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd2913d9-e59b-4b4a-b10a-a324e0c62f25-cert\") pod \"llmisvc-controller-manager-6466c554d8-28jss\" (UID: \"cd2913d9-e59b-4b4a-b10a-a324e0c62f25\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-28jss" Nov 02 21:16:57.893428 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:57.893362 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q782l\" (UniqueName: \"kubernetes.io/projected/cd2913d9-e59b-4b4a-b10a-a324e0c62f25-kube-api-access-q782l\") pod \"llmisvc-controller-manager-6466c554d8-28jss\" (UID: \"cd2913d9-e59b-4b4a-b10a-a324e0c62f25\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-28jss" Nov 02 21:16:57.956409 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:57.956379 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-28jss" Nov 02 21:16:58.325392 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:58.325357 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-28jss"] Nov 02 21:16:58.329155 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:16:58.329088 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd2913d9_e59b_4b4a_b10a_a324e0c62f25.slice/crio-a9065120c13dcfb4f25c35633000c4643b25c8502890b54d45b5c3b492c2db54 WatchSource:0}: Error finding container a9065120c13dcfb4f25c35633000c4643b25c8502890b54d45b5c3b492c2db54: Status 404 returned error can't find the container with id a9065120c13dcfb4f25c35633000c4643b25c8502890b54d45b5c3b492c2db54 Nov 02 21:16:58.468617 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:16:58.468579 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:16:58.486117 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:58.486084 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-28jss" event={"ID":"cd2913d9-e59b-4b4a-b10a-a324e0c62f25","Type":"ContainerStarted","Data":"a9065120c13dcfb4f25c35633000c4643b25c8502890b54d45b5c3b492c2db54"} Nov 02 21:16:59.492582 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:59.492486 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-28jss" event={"ID":"cd2913d9-e59b-4b4a-b10a-a324e0c62f25","Type":"ContainerStarted","Data":"06e29b23896bc0ed08c21a573850bf29f806652f9a81c11dce0b8c4fd7560afb"} Nov 02 21:16:59.492582 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:59.492573 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-28jss" Nov 02 21:16:59.598358 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:16:59.598302 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-28jss" podStartSLOduration=1.7914073130000001 podStartE2EDuration="2.598285678s" totalImagesPullingTime="806.878365ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:16:59.597651448 +0000 UTC m=+512.037226171" watchObservedRunningTime="2026-11-02 21:16:59.598285678 +0000 UTC m=+512.037860400" Nov 02 21:17:20.329787 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:20.329726 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:17:28.468635 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:17:28.468594 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:17:30.499775 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:30.499744 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-28jss" Nov 02 21:17:30.630715 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:30.630682 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xbz5g"] Nov 02 21:17:30.630922 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:30.630898 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbz5g" podUID="87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4" containerName="manager" containerID="cri-o://88bcda2071216888d4bb995de86c97e9016d3bc8d26a4894bcbae6c6aacef975" gracePeriod=30 Nov 02 21:17:30.881757 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:30.881729 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbz5g" Nov 02 21:17:31.033558 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:31.033520 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4-cert\" (UniqueName: \"kubernetes.io/secret/87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4-cert\") pod \"87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4\" (UID: \"87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4\") " Nov 02 21:17:31.033727 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:31.033627 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4-kube-api-access-vgtfg\" (UniqueName: \"kubernetes.io/projected/87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4-kube-api-access-vgtfg\") pod \"87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4\" (UID: \"87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4\") " Nov 02 21:17:31.035685 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:31.035647 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4-cert" pod "87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4" (UID: "87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:17:31.035812 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:31.035684 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4-kube-api-access-vgtfg" pod "87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4" (UID: "87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4"). InnerVolumeSpecName "kube-api-access-vgtfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:17:31.134482 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:31.134398 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgtfg\" (UniqueName: \"kubernetes.io/projected/87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4-kube-api-access-vgtfg\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:31.134482 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:31.134430 4935 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:31.658418 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:31.658382 4935 generic.go:356] "Generic (PLEG): container finished" podID="87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4" containerID="88bcda2071216888d4bb995de86c97e9016d3bc8d26a4894bcbae6c6aacef975" exitCode=0 Nov 02 21:17:31.658933 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:31.658429 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbz5g" event={"ID":"87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4","Type":"ContainerDied","Data":"88bcda2071216888d4bb995de86c97e9016d3bc8d26a4894bcbae6c6aacef975"} Nov 02 21:17:31.658933 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:31.658450 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbz5g" Nov 02 21:17:31.658933 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:31.658480 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xbz5g" event={"ID":"87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4","Type":"ContainerDied","Data":"66de1ce0a0af90e57f13459e307dd59b7cd84e11eac6e24b1789d35e681727ba"} Nov 02 21:17:31.658933 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:31.658512 4935 scope.go:122] "RemoveContainer" containerID="88bcda2071216888d4bb995de86c97e9016d3bc8d26a4894bcbae6c6aacef975" Nov 02 21:17:31.668243 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:31.668217 4935 scope.go:122] "RemoveContainer" containerID="88bcda2071216888d4bb995de86c97e9016d3bc8d26a4894bcbae6c6aacef975" Nov 02 21:17:31.668538 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:17:31.668514 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88bcda2071216888d4bb995de86c97e9016d3bc8d26a4894bcbae6c6aacef975\": container with ID starting with 88bcda2071216888d4bb995de86c97e9016d3bc8d26a4894bcbae6c6aacef975 not found: ID does not exist" containerID="88bcda2071216888d4bb995de86c97e9016d3bc8d26a4894bcbae6c6aacef975" Nov 02 21:17:31.668618 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:31.668551 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88bcda2071216888d4bb995de86c97e9016d3bc8d26a4894bcbae6c6aacef975"} err="failed to get container status \"88bcda2071216888d4bb995de86c97e9016d3bc8d26a4894bcbae6c6aacef975\": rpc error: code = NotFound desc = could not find container \"88bcda2071216888d4bb995de86c97e9016d3bc8d26a4894bcbae6c6aacef975\": container with ID starting with 88bcda2071216888d4bb995de86c97e9016d3bc8d26a4894bcbae6c6aacef975 not found: ID does not exist" Nov 02 21:17:31.761637 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:31.761575 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:17:31.812301 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:31.812269 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xbz5g"] Nov 02 21:17:31.815665 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:31.815641 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xbz5g"] Nov 02 21:17:32.097354 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:32.097322 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:17:33.779391 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:33.776371 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4" path="/var/lib/kubelet/pods/87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4/volumes" Nov 02 21:17:39.622780 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:39.622738 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:17:39.845026 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:39.844996 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:17:51.252438 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:51.252395 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:17:53.775102 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:53.775068 4935 generic.go:356] "Generic (PLEG): container finished" podID="7e1298eb-5e6b-4e00-9269-d88a130d72c7" containerID="d04c62f90ac285517935481f026be68907815ec8e7c7f6c523bb29774be7ebc6" exitCode=1 Nov 02 21:17:53.775497 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:53.775159 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d468bc7cc-hj64v" event={"ID":"7e1298eb-5e6b-4e00-9269-d88a130d72c7","Type":"ContainerDied","Data":"d04c62f90ac285517935481f026be68907815ec8e7c7f6c523bb29774be7ebc6"} Nov 02 21:17:53.775539 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:53.775495 4935 scope.go:122] "RemoveContainer" containerID="d04c62f90ac285517935481f026be68907815ec8e7c7f6c523bb29774be7ebc6" Nov 02 21:17:55.788706 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:55.788669 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d468bc7cc-hj64v" event={"ID":"7e1298eb-5e6b-4e00-9269-d88a130d72c7","Type":"ContainerStarted","Data":"cb34b6a642ef2fb34cda353d348de41b942b42e57a3fe07f4f9fd9efe39ce643"} Nov 02 21:17:55.789098 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:17:55.788882 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d468bc7cc-hj64v" Nov 02 21:17:58.468296 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:17:58.468261 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:18:01.833258 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:18:01.833228 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:02.310833 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:18:02.310798 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:03.459287 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:18:03.459249 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:03.686394 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:18:03.686359 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:06.795972 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:18:06.795939 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d468bc7cc-hj64v" Nov 02 21:18:13.541990 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:18:13.541897 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:15.119339 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:18:15.119306 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:15.365147 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:18:15.365094 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:19.448360 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:18:19.448329 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:20.650586 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:18:20.650552 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:23.110499 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:18:23.110460 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:27.264365 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:18:27.264303 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:27.884389 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:18:27.884353 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 21:18:27.884569 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:18:27.884465 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 21:18:27.894538 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:18:27.894511 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 21:18:27.894727 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:18:27.894702 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 21:18:28.468443 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:18:28.468408 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:18:31.251734 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:18:31.251692 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:35.213593 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:18:35.213553 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:40.773891 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:18:40.773853 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:44.222499 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:18:44.222444 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:18:58.468482 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:18:58.468437 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:18:59.977365 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:18:59.977330 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn"] Nov 02 21:18:59.977742 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:18:59.977709 4935 cpu_manager.go:414] "containerMap: removing container" podUID="87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4" containerName="manager" Nov 02 21:18:59.977742 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:18:59.977719 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4" containerName="manager" Nov 02 21:18:59.977815 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:18:59.977811 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="87f4d61d-3160-4910-b0c4-5f4b6eb5a6b4" containerName="manager" Nov 02 21:18:59.981444 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:18:59.981417 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.044776 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.044739 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2gjqn\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.044776 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.044780 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/430fb50a-4cbe-4b91-9e00-7484df623aec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2gjqn\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.045021 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.044807 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2gjqn\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.045021 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.044877 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v56zq\" (UniqueName: \"kubernetes.io/projected/430fb50a-4cbe-4b91-9e00-7484df623aec-kube-api-access-v56zq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2gjqn\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.045021 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.044985 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2gjqn\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.045176 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.045038 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/430fb50a-4cbe-4b91-9e00-7484df623aec-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2gjqn\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.045176 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.045112 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/430fb50a-4cbe-4b91-9e00-7484df623aec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2gjqn\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.045253 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.045223 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2gjqn\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.045253 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.045243 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2gjqn\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.080177 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.080147 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn"] Nov 02 21:19:00.146026 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.145992 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2gjqn\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.146026 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.146027 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2gjqn\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.146312 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.146048 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2gjqn\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.146312 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.146112 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/430fb50a-4cbe-4b91-9e00-7484df623aec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2gjqn\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.146312 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.146188 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2gjqn\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.146312 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.146207 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v56zq\" (UniqueName: \"kubernetes.io/projected/430fb50a-4cbe-4b91-9e00-7484df623aec-kube-api-access-v56zq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2gjqn\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.146312 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.146224 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2gjqn\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.146312 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.146251 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/430fb50a-4cbe-4b91-9e00-7484df623aec-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2gjqn\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.146312 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.146297 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/430fb50a-4cbe-4b91-9e00-7484df623aec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2gjqn\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.146686 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.146527 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2gjqn\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.146686 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.146655 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2gjqn\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.146856 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.146691 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2gjqn\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.146856 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.146799 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2gjqn\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.146998 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.146977 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/430fb50a-4cbe-4b91-9e00-7484df623aec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2gjqn\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.148434 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.148414 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2gjqn\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.148644 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.148627 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/430fb50a-4cbe-4b91-9e00-7484df623aec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2gjqn\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.199343 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.199303 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/430fb50a-4cbe-4b91-9e00-7484df623aec-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2gjqn\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.199343 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.199332 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v56zq\" (UniqueName: \"kubernetes.io/projected/430fb50a-4cbe-4b91-9e00-7484df623aec-kube-api-access-v56zq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2gjqn\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.294515 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.294485 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:00.711212 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.711185 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn"] Nov 02 21:19:00.713315 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:19:00.713282 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod430fb50a_4cbe_4b91_9e00_7484df623aec.slice/crio-8050284d67b8ae95dda03f5d8f9ab54013baee9b34b13b05190fa504c644f73c WatchSource:0}: Error finding container 8050284d67b8ae95dda03f5d8f9ab54013baee9b34b13b05190fa504c644f73c: Status 404 returned error can't find the container with id 8050284d67b8ae95dda03f5d8f9ab54013baee9b34b13b05190fa504c644f73c Nov 02 21:19:00.715507 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.715472 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:19:00.715624 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.715538 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:19:00.715624 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:00.715568 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:19:01.145249 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:01.145205 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" event={"ID":"430fb50a-4cbe-4b91-9e00-7484df623aec","Type":"ContainerStarted","Data":"1888087af1eb35a768641da3e93ecbf331fbbfc1f2d02ea8868bc75611aff1ca"} Nov 02 21:19:01.145759 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:01.145261 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" event={"ID":"430fb50a-4cbe-4b91-9e00-7484df623aec","Type":"ContainerStarted","Data":"8050284d67b8ae95dda03f5d8f9ab54013baee9b34b13b05190fa504c644f73c"} Nov 02 21:19:01.295457 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:01.295420 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:01.296983 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:01.296962 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2gjqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 02 21:19:01.297046 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:01.297007 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" podUID="430fb50a-4cbe-4b91-9e00-7484df623aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 02 21:19:01.300834 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:01.300778 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" podStartSLOduration=2.300761855 podStartE2EDuration="2.300761855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:19:01.299240564 +0000 UTC m=+633.738815286" watchObservedRunningTime="2026-11-02 21:19:01.300761855 +0000 UTC m=+633.740336580" Nov 02 21:19:02.295448 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:02.295411 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2gjqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 02 21:19:02.295863 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:02.295469 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" podUID="430fb50a-4cbe-4b91-9e00-7484df623aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 02 21:19:03.295690 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:03.295655 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2gjqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 02 21:19:03.296077 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:03.295706 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" podUID="430fb50a-4cbe-4b91-9e00-7484df623aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 02 21:19:04.295366 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:04.295330 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2gjqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 02 21:19:04.295543 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:04.295386 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" podUID="430fb50a-4cbe-4b91-9e00-7484df623aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 02 21:19:05.295794 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:05.295756 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2gjqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 02 21:19:05.296217 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:05.295812 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" podUID="430fb50a-4cbe-4b91-9e00-7484df623aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 02 21:19:06.295438 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:06.295400 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2gjqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 02 21:19:06.295707 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:06.295454 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" podUID="430fb50a-4cbe-4b91-9e00-7484df623aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 02 21:19:07.295735 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:07.295697 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2gjqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 02 21:19:07.296202 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:07.295753 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" podUID="430fb50a-4cbe-4b91-9e00-7484df623aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 02 21:19:09.043670 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:09.043635 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:09.044099 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:09.043964 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:09.044764 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:09.044741 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:19:12.828839 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:12.828807 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:13.057911 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:13.057882 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:14.316515 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:14.316485 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:14.520519 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:14.520487 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b548fd7c-bxrbs"] Nov 02 21:19:14.520693 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:14.520673 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5b548fd7c-bxrbs" podUID="9ae05787-1ce2-40c7-ad94-e18443194248" containerName="authorino" containerID="cri-o://29a17ef97678f034737c8839cc92611aa0e8d3c1d212d1641135eba266ca0bc7" gracePeriod=30 Nov 02 21:19:14.547757 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:14.547729 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:14.782967 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:14.782940 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b548fd7c-bxrbs" Nov 02 21:19:14.882636 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:14.882609 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ae05787-1ce2-40c7-ad94-e18443194248-tls-cert\" (UniqueName: \"kubernetes.io/secret/9ae05787-1ce2-40c7-ad94-e18443194248-tls-cert\") pod \"9ae05787-1ce2-40c7-ad94-e18443194248\" (UID: \"9ae05787-1ce2-40c7-ad94-e18443194248\") " Nov 02 21:19:14.882792 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:14.882704 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ae05787-1ce2-40c7-ad94-e18443194248-kube-api-access-lqx7d\" (UniqueName: \"kubernetes.io/projected/9ae05787-1ce2-40c7-ad94-e18443194248-kube-api-access-lqx7d\") pod \"9ae05787-1ce2-40c7-ad94-e18443194248\" (UID: \"9ae05787-1ce2-40c7-ad94-e18443194248\") " Nov 02 21:19:14.884616 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:14.884582 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae05787-1ce2-40c7-ad94-e18443194248-kube-api-access-lqx7d" pod "9ae05787-1ce2-40c7-ad94-e18443194248" (UID: "9ae05787-1ce2-40c7-ad94-e18443194248"). InnerVolumeSpecName "kube-api-access-lqx7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:19:14.895379 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:14.895342 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ae05787-1ce2-40c7-ad94-e18443194248-tls-cert" pod "9ae05787-1ce2-40c7-ad94-e18443194248" (UID: "9ae05787-1ce2-40c7-ad94-e18443194248"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:19:14.983635 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:14.983602 4935 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9ae05787-1ce2-40c7-ad94-e18443194248-tls-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:19:14.983635 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:14.983630 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqx7d\" (UniqueName: \"kubernetes.io/projected/9ae05787-1ce2-40c7-ad94-e18443194248-kube-api-access-lqx7d\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:19:15.226843 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:15.226751 4935 generic.go:356] "Generic (PLEG): container finished" podID="9ae05787-1ce2-40c7-ad94-e18443194248" containerID="29a17ef97678f034737c8839cc92611aa0e8d3c1d212d1641135eba266ca0bc7" exitCode=0 Nov 02 21:19:15.226843 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:15.226799 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b548fd7c-bxrbs" Nov 02 21:19:15.227054 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:15.226843 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b548fd7c-bxrbs" event={"ID":"9ae05787-1ce2-40c7-ad94-e18443194248","Type":"ContainerDied","Data":"29a17ef97678f034737c8839cc92611aa0e8d3c1d212d1641135eba266ca0bc7"} Nov 02 21:19:15.227054 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:15.226894 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b548fd7c-bxrbs" event={"ID":"9ae05787-1ce2-40c7-ad94-e18443194248","Type":"ContainerDied","Data":"a2baec864efb778ed2790afd146b99ef0bf8efdc13ed79647c60c9868057f5b3"} Nov 02 21:19:15.227054 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:15.226918 4935 scope.go:122] "RemoveContainer" containerID="29a17ef97678f034737c8839cc92611aa0e8d3c1d212d1641135eba266ca0bc7" Nov 02 21:19:15.236269 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:15.236250 4935 scope.go:122] "RemoveContainer" containerID="29a17ef97678f034737c8839cc92611aa0e8d3c1d212d1641135eba266ca0bc7" Nov 02 21:19:15.236544 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:19:15.236525 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29a17ef97678f034737c8839cc92611aa0e8d3c1d212d1641135eba266ca0bc7\": container with ID starting with 29a17ef97678f034737c8839cc92611aa0e8d3c1d212d1641135eba266ca0bc7 not found: ID does not exist" containerID="29a17ef97678f034737c8839cc92611aa0e8d3c1d212d1641135eba266ca0bc7" Nov 02 21:19:15.236609 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:15.236556 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29a17ef97678f034737c8839cc92611aa0e8d3c1d212d1641135eba266ca0bc7"} err="failed to get container status \"29a17ef97678f034737c8839cc92611aa0e8d3c1d212d1641135eba266ca0bc7\": rpc error: code = NotFound desc = could not find container \"29a17ef97678f034737c8839cc92611aa0e8d3c1d212d1641135eba266ca0bc7\": container with ID starting with 29a17ef97678f034737c8839cc92611aa0e8d3c1d212d1641135eba266ca0bc7 not found: ID does not exist" Nov 02 21:19:15.387106 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:15.387074 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b548fd7c-bxrbs"] Nov 02 21:19:15.433496 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:15.433464 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5b548fd7c-bxrbs"] Nov 02 21:19:15.774930 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:15.774893 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ae05787-1ce2-40c7-ad94-e18443194248" path="/var/lib/kubelet/pods/9ae05787-1ce2-40c7-ad94-e18443194248/volumes" Nov 02 21:19:20.189811 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:20.189771 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:20.976589 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:20.976558 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:28.468740 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:19:28.468699 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:19:43.216677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:43.216597 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:43.481016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:43.480930 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:43.763899 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:43.763864 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:43.991696 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:43.991660 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:44.226641 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:44.226557 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:44.450661 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:44.450625 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:44.675565 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:44.675525 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:44.906882 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:44.906848 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:45.136689 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:45.136652 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:46.441608 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:46.441571 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:46.662653 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:46.662617 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:46.882390 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:46.882353 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:47.923009 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:47.922978 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:48.254691 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:48.254611 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:48.584014 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:48.583980 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:48.815031 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:48.814998 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:49.657645 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:49.657606 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:49.886232 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:49.886203 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:50.113241 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:50.113199 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:50.405160 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:50.405066 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:50.971264 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:50.971232 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:51.217526 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:51.217492 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:52.051628 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:52.051590 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:52.275675 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:52.275639 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:54.090652 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:54.090593 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:54.323522 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:54.323467 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:54.693934 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:54.693897 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:54.917501 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:54.917471 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:56.520656 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:56.520624 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:56.755857 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:56.755819 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:58.248703 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:58.248667 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:58.470980 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:19:58.470946 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:19:58.864784 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:58.864750 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:59.135401 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:59.135317 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:19:59.450764 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:19:59.450679 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:00.135564 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:00.135528 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g"] Nov 02 21:20:00.136021 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:00.136002 4935 cpu_manager.go:414] "containerMap: removing container" podUID="9ae05787-1ce2-40c7-ad94-e18443194248" containerName="authorino" Nov 02 21:20:00.136021 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:00.136025 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ae05787-1ce2-40c7-ad94-e18443194248" containerName="authorino" Nov 02 21:20:00.136273 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:00.136193 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ae05787-1ce2-40c7-ad94-e18443194248" containerName="authorino" Nov 02 21:20:00.139410 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:00.139384 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g" Nov 02 21:20:00.188059 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:00.188025 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd46849e-e2c9-427a-b93b-839dd480a585-maas-api-tls\") pod \"maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g\" (UID: \"dd46849e-e2c9-427a-b93b-839dd480a585\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g" Nov 02 21:20:00.188249 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:00.188067 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd46849e-e2c9-427a-b93b-839dd480a585-metrics-certs\") pod \"maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g\" (UID: \"dd46849e-e2c9-427a-b93b-839dd480a585\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g" Nov 02 21:20:00.188249 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:00.188090 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv89g\" (UniqueName: \"kubernetes.io/projected/dd46849e-e2c9-427a-b93b-839dd480a585-kube-api-access-bv89g\") pod \"maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g\" (UID: \"dd46849e-e2c9-427a-b93b-839dd480a585\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g" Nov 02 21:20:00.188337 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:00.188261 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd46849e-e2c9-427a-b93b-839dd480a585-openshift-service-ca\") pod \"maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g\" (UID: \"dd46849e-e2c9-427a-b93b-839dd480a585\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g" Nov 02 21:20:00.189966 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:00.189752 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-5478da31-serving-cert\"" Nov 02 21:20:00.238245 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:00.238214 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g"] Nov 02 21:20:00.288722 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:00.288688 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd46849e-e2c9-427a-b93b-839dd480a585-maas-api-tls\") pod \"maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g\" (UID: \"dd46849e-e2c9-427a-b93b-839dd480a585\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g" Nov 02 21:20:00.288986 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:00.288733 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd46849e-e2c9-427a-b93b-839dd480a585-metrics-certs\") pod \"maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g\" (UID: \"dd46849e-e2c9-427a-b93b-839dd480a585\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g" Nov 02 21:20:00.288986 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:00.288751 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv89g\" (UniqueName: \"kubernetes.io/projected/dd46849e-e2c9-427a-b93b-839dd480a585-kube-api-access-bv89g\") pod \"maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g\" (UID: \"dd46849e-e2c9-427a-b93b-839dd480a585\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g" Nov 02 21:20:00.288986 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:00.288774 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd46849e-e2c9-427a-b93b-839dd480a585-openshift-service-ca\") pod \"maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g\" (UID: \"dd46849e-e2c9-427a-b93b-839dd480a585\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g" Nov 02 21:20:00.288986 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:20:00.288862 4935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-serving-cert: secret "maas-api-e2e-migrate-5478da31-serving-cert" not found Nov 02 21:20:00.288986 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:20:00.288954 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd46849e-e2c9-427a-b93b-839dd480a585-maas-api-tls podName:dd46849e-e2c9-427a-b93b-839dd480a585 nodeName:}" failed. No retries permitted until 2026-11-02 21:20:00.788928479 +0000 UTC m=+693.228503181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dd46849e-e2c9-427a-b93b-839dd480a585-maas-api-tls") pod "maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g" (UID: "dd46849e-e2c9-427a-b93b-839dd480a585") : secret "maas-api-e2e-migrate-5478da31-serving-cert" not found Nov 02 21:20:00.289554 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:00.289533 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd46849e-e2c9-427a-b93b-839dd480a585-openshift-service-ca\") pod \"maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g\" (UID: \"dd46849e-e2c9-427a-b93b-839dd480a585\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g" Nov 02 21:20:00.291598 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:00.291573 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd46849e-e2c9-427a-b93b-839dd480a585-metrics-certs\") pod \"maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g\" (UID: \"dd46849e-e2c9-427a-b93b-839dd480a585\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g" Nov 02 21:20:00.344876 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:00.344846 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv89g\" (UniqueName: \"kubernetes.io/projected/dd46849e-e2c9-427a-b93b-839dd480a585-kube-api-access-bv89g\") pod \"maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g\" (UID: \"dd46849e-e2c9-427a-b93b-839dd480a585\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g" Nov 02 21:20:00.475564 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:00.475487 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:00.694402 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:00.694347 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:00.793728 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:00.793695 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd46849e-e2c9-427a-b93b-839dd480a585-maas-api-tls\") pod \"maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g\" (UID: \"dd46849e-e2c9-427a-b93b-839dd480a585\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g" Nov 02 21:20:00.796186 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:00.796157 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd46849e-e2c9-427a-b93b-839dd480a585-maas-api-tls\") pod \"maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g\" (UID: \"dd46849e-e2c9-427a-b93b-839dd480a585\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g" Nov 02 21:20:01.052264 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:01.052185 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g" Nov 02 21:20:01.335880 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:01.335804 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:01.423451 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:01.423406 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g"] Nov 02 21:20:01.427206 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:20:01.427172 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd46849e_e2c9_427a_b93b_839dd480a585.slice/crio-ee27e78cd938a657b8f0166b2a3fee71dcb670ba258d30cdd8ee3596941a646e WatchSource:0}: Error finding container ee27e78cd938a657b8f0166b2a3fee71dcb670ba258d30cdd8ee3596941a646e: Status 404 returned error can't find the container with id ee27e78cd938a657b8f0166b2a3fee71dcb670ba258d30cdd8ee3596941a646e Nov 02 21:20:01.428720 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:01.428700 4935 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:20:01.481827 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:01.481790 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g" event={"ID":"dd46849e-e2c9-427a-b93b-839dd480a585","Type":"ContainerStarted","Data":"ee27e78cd938a657b8f0166b2a3fee71dcb670ba258d30cdd8ee3596941a646e"} Nov 02 21:20:01.554173 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:01.554109 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:02.488328 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:02.488293 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g" event={"ID":"dd46849e-e2c9-427a-b93b-839dd480a585","Type":"ContainerStarted","Data":"5cf66975cb67c06b52c1326606e816a40a5cdb95959ea3e107cc911ef85699e9"} Nov 02 21:20:02.488794 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:02.488414 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g" Nov 02 21:20:02.597875 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:02.597761 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g" podStartSLOduration=1.729602994 podStartE2EDuration="2.597746745s" totalImagesPullingTime="868.143751ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:20:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:20:02.595238884 +0000 UTC m=+695.034813605" watchObservedRunningTime="2026-11-02 21:20:02.597746745 +0000 UTC m=+695.037321466" Nov 02 21:20:05.595467 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:05.595432 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj"] Nov 02 21:20:05.598982 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:05.598965 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj" Nov 02 21:20:05.647327 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:05.647300 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-84de7f3a-serving-cert\"" Nov 02 21:20:05.696949 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:05.696913 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj"] Nov 02 21:20:05.738996 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:05.738962 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bpts\" (UniqueName: \"kubernetes.io/projected/51116ad1-b97f-42fa-80d2-a10cd0af3e45-kube-api-access-2bpts\") pod \"maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj\" (UID: \"51116ad1-b97f-42fa-80d2-a10cd0af3e45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj" Nov 02 21:20:05.738996 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:05.738997 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51116ad1-b97f-42fa-80d2-a10cd0af3e45-openshift-service-ca\") pod \"maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj\" (UID: \"51116ad1-b97f-42fa-80d2-a10cd0af3e45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj" Nov 02 21:20:05.739219 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:05.739041 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51116ad1-b97f-42fa-80d2-a10cd0af3e45-maas-api-tls\") pod \"maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj\" (UID: \"51116ad1-b97f-42fa-80d2-a10cd0af3e45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj" Nov 02 21:20:05.739219 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:05.739186 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51116ad1-b97f-42fa-80d2-a10cd0af3e45-metrics-certs\") pod \"maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj\" (UID: \"51116ad1-b97f-42fa-80d2-a10cd0af3e45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj" Nov 02 21:20:05.839976 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:05.839945 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51116ad1-b97f-42fa-80d2-a10cd0af3e45-maas-api-tls\") pod \"maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj\" (UID: \"51116ad1-b97f-42fa-80d2-a10cd0af3e45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj" Nov 02 21:20:05.840191 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:05.840065 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51116ad1-b97f-42fa-80d2-a10cd0af3e45-metrics-certs\") pod \"maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj\" (UID: \"51116ad1-b97f-42fa-80d2-a10cd0af3e45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj" Nov 02 21:20:05.840272 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:05.840207 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bpts\" (UniqueName: \"kubernetes.io/projected/51116ad1-b97f-42fa-80d2-a10cd0af3e45-kube-api-access-2bpts\") pod \"maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj\" (UID: \"51116ad1-b97f-42fa-80d2-a10cd0af3e45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj" Nov 02 21:20:05.840272 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:05.840234 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51116ad1-b97f-42fa-80d2-a10cd0af3e45-openshift-service-ca\") pod \"maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj\" (UID: \"51116ad1-b97f-42fa-80d2-a10cd0af3e45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj" Nov 02 21:20:05.840605 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:20:05.840573 4935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-serving-cert: secret "maas-api-e2e-ait-84de7f3a-serving-cert" not found Nov 02 21:20:05.840747 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:20:05.840682 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51116ad1-b97f-42fa-80d2-a10cd0af3e45-maas-api-tls podName:51116ad1-b97f-42fa-80d2-a10cd0af3e45 nodeName:}" failed. No retries permitted until 2026-11-02 21:20:06.340660025 +0000 UTC m=+698.780234725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/51116ad1-b97f-42fa-80d2-a10cd0af3e45-maas-api-tls") pod "maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj" (UID: "51116ad1-b97f-42fa-80d2-a10cd0af3e45") : secret "maas-api-e2e-ait-84de7f3a-serving-cert" not found Nov 02 21:20:05.840903 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:05.840883 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51116ad1-b97f-42fa-80d2-a10cd0af3e45-openshift-service-ca\") pod \"maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj\" (UID: \"51116ad1-b97f-42fa-80d2-a10cd0af3e45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj" Nov 02 21:20:05.843222 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:05.843199 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51116ad1-b97f-42fa-80d2-a10cd0af3e45-metrics-certs\") pod \"maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj\" (UID: \"51116ad1-b97f-42fa-80d2-a10cd0af3e45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj" Nov 02 21:20:05.891727 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:05.891644 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:05.893983 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:05.893958 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bpts\" (UniqueName: \"kubernetes.io/projected/51116ad1-b97f-42fa-80d2-a10cd0af3e45-kube-api-access-2bpts\") pod \"maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj\" (UID: \"51116ad1-b97f-42fa-80d2-a10cd0af3e45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj" Nov 02 21:20:06.114064 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:06.114023 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:06.344922 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:06.344888 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51116ad1-b97f-42fa-80d2-a10cd0af3e45-maas-api-tls\") pod \"maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj\" (UID: \"51116ad1-b97f-42fa-80d2-a10cd0af3e45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj" Nov 02 21:20:06.347568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:06.347544 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51116ad1-b97f-42fa-80d2-a10cd0af3e45-maas-api-tls\") pod \"maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj\" (UID: \"51116ad1-b97f-42fa-80d2-a10cd0af3e45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj" Nov 02 21:20:06.512623 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:06.512580 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj" Nov 02 21:20:06.888429 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:06.888404 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj"] Nov 02 21:20:06.891008 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:20:06.890978 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51116ad1_b97f_42fa_80d2_a10cd0af3e45.slice/crio-965b52b78953cdb0c59d93e21edc538050d5ed06a5e14298aa37a984943d22ec WatchSource:0}: Error finding container 965b52b78953cdb0c59d93e21edc538050d5ed06a5e14298aa37a984943d22ec: Status 404 returned error can't find the container with id 965b52b78953cdb0c59d93e21edc538050d5ed06a5e14298aa37a984943d22ec Nov 02 21:20:07.520479 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:07.520444 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj" event={"ID":"51116ad1-b97f-42fa-80d2-a10cd0af3e45","Type":"ContainerStarted","Data":"965b52b78953cdb0c59d93e21edc538050d5ed06a5e14298aa37a984943d22ec"} Nov 02 21:20:08.499881 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:08.499854 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g" Nov 02 21:20:08.528392 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:08.528359 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj" event={"ID":"51116ad1-b97f-42fa-80d2-a10cd0af3e45","Type":"ContainerStarted","Data":"9942ecaf6ab6ee398c698f27809d85e75d3fce8891388b5074cf6d1a7f961a06"} Nov 02 21:20:08.528550 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:08.528437 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj" Nov 02 21:20:08.718481 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:08.718389 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj" podStartSLOduration=2.7563291420000002 podStartE2EDuration="3.718372338s" totalImagesPullingTime="962.043196ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:20:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:20:08.717527655 +0000 UTC m=+701.157102378" watchObservedRunningTime="2026-11-02 21:20:08.718372338 +0000 UTC m=+701.157947061" Nov 02 21:20:10.253920 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:10.253880 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:10.484377 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:10.484345 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:11.584637 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:11.584605 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:11.810738 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:11.810704 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:14.539051 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:14.539018 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj" Nov 02 21:20:16.845588 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:16.845550 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:21.327854 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:21.327796 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:21.596881 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:21.596785 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j"] Nov 02 21:20:21.601910 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:21.601884 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j" Nov 02 21:20:21.652352 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:21.652325 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j"] Nov 02 21:20:21.685609 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:21.685580 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deca8e64-4310-4e47-a1bb-b037781009c8-maas-api-tls\") pod \"maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j\" (UID: \"deca8e64-4310-4e47-a1bb-b037781009c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j" Nov 02 21:20:21.685737 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:21.685620 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deca8e64-4310-4e47-a1bb-b037781009c8-openshift-service-ca\") pod \"maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j\" (UID: \"deca8e64-4310-4e47-a1bb-b037781009c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j" Nov 02 21:20:21.685737 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:21.685649 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deca8e64-4310-4e47-a1bb-b037781009c8-metrics-certs\") pod \"maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j\" (UID: \"deca8e64-4310-4e47-a1bb-b037781009c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j" Nov 02 21:20:21.685811 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:21.685785 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59wdc\" (UniqueName: \"kubernetes.io/projected/deca8e64-4310-4e47-a1bb-b037781009c8-kube-api-access-59wdc\") pod \"maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j\" (UID: \"deca8e64-4310-4e47-a1bb-b037781009c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j" Nov 02 21:20:21.696157 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:21.696120 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-54a47302-serving-cert\"" Nov 02 21:20:21.786934 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:21.786891 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deca8e64-4310-4e47-a1bb-b037781009c8-maas-api-tls\") pod \"maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j\" (UID: \"deca8e64-4310-4e47-a1bb-b037781009c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j" Nov 02 21:20:21.787108 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:21.786954 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deca8e64-4310-4e47-a1bb-b037781009c8-openshift-service-ca\") pod \"maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j\" (UID: \"deca8e64-4310-4e47-a1bb-b037781009c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j" Nov 02 21:20:21.787108 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:21.786984 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deca8e64-4310-4e47-a1bb-b037781009c8-metrics-certs\") pod \"maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j\" (UID: \"deca8e64-4310-4e47-a1bb-b037781009c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j" Nov 02 21:20:21.787108 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:21.787009 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59wdc\" (UniqueName: \"kubernetes.io/projected/deca8e64-4310-4e47-a1bb-b037781009c8-kube-api-access-59wdc\") pod \"maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j\" (UID: \"deca8e64-4310-4e47-a1bb-b037781009c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j" Nov 02 21:20:21.787108 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:20:21.787034 4935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-serving-cert: secret "maas-api-e2e-derive-54a47302-serving-cert" not found Nov 02 21:20:21.787371 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:20:21.787113 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deca8e64-4310-4e47-a1bb-b037781009c8-maas-api-tls podName:deca8e64-4310-4e47-a1bb-b037781009c8 nodeName:}" failed. No retries permitted until 2026-11-02 21:20:22.287095913 +0000 UTC m=+714.726670613 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/deca8e64-4310-4e47-a1bb-b037781009c8-maas-api-tls") pod "maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j" (UID: "deca8e64-4310-4e47-a1bb-b037781009c8") : secret "maas-api-e2e-derive-54a47302-serving-cert" not found Nov 02 21:20:21.787700 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:21.787677 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deca8e64-4310-4e47-a1bb-b037781009c8-openshift-service-ca\") pod \"maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j\" (UID: \"deca8e64-4310-4e47-a1bb-b037781009c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j" Nov 02 21:20:21.789349 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:21.789330 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deca8e64-4310-4e47-a1bb-b037781009c8-metrics-certs\") pod \"maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j\" (UID: \"deca8e64-4310-4e47-a1bb-b037781009c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j" Nov 02 21:20:21.840621 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:21.840591 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59wdc\" (UniqueName: \"kubernetes.io/projected/deca8e64-4310-4e47-a1bb-b037781009c8-kube-api-access-59wdc\") pod \"maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j\" (UID: \"deca8e64-4310-4e47-a1bb-b037781009c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j" Nov 02 21:20:21.889078 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:21.888998 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:22.108598 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:22.108563 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:22.291588 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:22.291537 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deca8e64-4310-4e47-a1bb-b037781009c8-maas-api-tls\") pod \"maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j\" (UID: \"deca8e64-4310-4e47-a1bb-b037781009c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j" Nov 02 21:20:22.293938 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:22.293908 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deca8e64-4310-4e47-a1bb-b037781009c8-maas-api-tls\") pod \"maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j\" (UID: \"deca8e64-4310-4e47-a1bb-b037781009c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j" Nov 02 21:20:22.341357 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:22.341327 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:22.514627 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:22.514589 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j" Nov 02 21:20:22.883385 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:22.883361 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j"] Nov 02 21:20:22.886563 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:20:22.886441 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeca8e64_4310_4e47_a1bb_b037781009c8.slice/crio-eb4c267d3cc1796526d465ffa8d1e6307aacd323091d196adacc0a78ce6fb54e WatchSource:0}: Error finding container eb4c267d3cc1796526d465ffa8d1e6307aacd323091d196adacc0a78ce6fb54e: Status 404 returned error can't find the container with id eb4c267d3cc1796526d465ffa8d1e6307aacd323091d196adacc0a78ce6fb54e Nov 02 21:20:23.613846 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:23.613811 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j" event={"ID":"deca8e64-4310-4e47-a1bb-b037781009c8","Type":"ContainerStarted","Data":"eb4c267d3cc1796526d465ffa8d1e6307aacd323091d196adacc0a78ce6fb54e"} Nov 02 21:20:24.620573 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:24.620530 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j" event={"ID":"deca8e64-4310-4e47-a1bb-b037781009c8","Type":"ContainerStarted","Data":"089a6a774a62f2a1547ea02b80a9714c92757d8bbe341e3754f56419bed2f529"} Nov 02 21:20:24.621048 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:24.620596 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j" Nov 02 21:20:24.728042 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:24.727992 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j" podStartSLOduration=2.846572305 podStartE2EDuration="3.727978539s" totalImagesPullingTime="881.406234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:20:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:20:24.727169995 +0000 UTC m=+717.166744716" watchObservedRunningTime="2026-11-02 21:20:24.727978539 +0000 UTC m=+717.167553255" Nov 02 21:20:27.141255 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:27.141206 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:28.470308 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:20:28.470266 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:20:30.631349 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:30.631320 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j" Nov 02 21:20:32.161775 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:32.161737 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:37.326585 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:37.326545 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:42.834343 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:42.834305 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:46.886606 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:46.886568 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:47.127848 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:47.127814 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:48.439331 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:48.439301 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:49.634872 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:49.634838 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:49.852667 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:49.852637 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:51.505390 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:51.505322 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:51.727158 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:51.727101 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:53.922580 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:53.922543 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:58.382027 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:58.381990 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:58.470507 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:20:58.470470 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:20:58.602743 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:58.602705 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:58.830740 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:58.830696 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:59.696877 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:59.696842 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g"] Nov 02 21:20:59.697269 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:59.697071 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g" podUID="dd46849e-e2c9-427a-b93b-839dd480a585" containerName="maas-api" containerID="cri-o://5cf66975cb67c06b52c1326606e816a40a5cdb95959ea3e107cc911ef85699e9" gracePeriod=30 Nov 02 21:20:59.885632 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:59.885551 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:20:59.971954 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:20:59.971927 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g" Nov 02 21:21:00.046037 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:00.046006 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd46849e-e2c9-427a-b93b-839dd480a585-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd46849e-e2c9-427a-b93b-839dd480a585-metrics-certs\") pod \"dd46849e-e2c9-427a-b93b-839dd480a585\" (UID: \"dd46849e-e2c9-427a-b93b-839dd480a585\") " Nov 02 21:21:00.046254 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:00.046076 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd46849e-e2c9-427a-b93b-839dd480a585-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd46849e-e2c9-427a-b93b-839dd480a585-openshift-service-ca\") pod \"dd46849e-e2c9-427a-b93b-839dd480a585\" (UID: \"dd46849e-e2c9-427a-b93b-839dd480a585\") " Nov 02 21:21:00.046254 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:00.046149 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd46849e-e2c9-427a-b93b-839dd480a585-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd46849e-e2c9-427a-b93b-839dd480a585-maas-api-tls\") pod \"dd46849e-e2c9-427a-b93b-839dd480a585\" (UID: \"dd46849e-e2c9-427a-b93b-839dd480a585\") " Nov 02 21:21:00.046254 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:00.046180 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd46849e-e2c9-427a-b93b-839dd480a585-kube-api-access-bv89g\" (UniqueName: \"kubernetes.io/projected/dd46849e-e2c9-427a-b93b-839dd480a585-kube-api-access-bv89g\") pod \"dd46849e-e2c9-427a-b93b-839dd480a585\" (UID: \"dd46849e-e2c9-427a-b93b-839dd480a585\") " Nov 02 21:21:00.046521 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:00.046492 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd46849e-e2c9-427a-b93b-839dd480a585-openshift-service-ca" pod "dd46849e-e2c9-427a-b93b-839dd480a585" (UID: "dd46849e-e2c9-427a-b93b-839dd480a585"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:21:00.048381 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:00.048343 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd46849e-e2c9-427a-b93b-839dd480a585-metrics-certs" pod "dd46849e-e2c9-427a-b93b-839dd480a585" (UID: "dd46849e-e2c9-427a-b93b-839dd480a585"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:21:00.048530 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:00.048381 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd46849e-e2c9-427a-b93b-839dd480a585-maas-api-tls" pod "dd46849e-e2c9-427a-b93b-839dd480a585" (UID: "dd46849e-e2c9-427a-b93b-839dd480a585"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:21:00.048530 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:00.048420 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd46849e-e2c9-427a-b93b-839dd480a585-kube-api-access-bv89g" pod "dd46849e-e2c9-427a-b93b-839dd480a585" (UID: "dd46849e-e2c9-427a-b93b-839dd480a585"). InnerVolumeSpecName "kube-api-access-bv89g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:21:00.147593 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:00.147553 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd46849e-e2c9-427a-b93b-839dd480a585-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:00.147593 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:00.147590 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd46849e-e2c9-427a-b93b-839dd480a585-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:00.147797 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:00.147606 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd46849e-e2c9-427a-b93b-839dd480a585-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:00.147797 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:00.147619 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv89g\" (UniqueName: \"kubernetes.io/projected/dd46849e-e2c9-427a-b93b-839dd480a585-kube-api-access-bv89g\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:00.266511 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:00.266476 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:00.569159 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:00.569025 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:00.821352 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:00.821273 4935 generic.go:356] "Generic (PLEG): container finished" podID="dd46849e-e2c9-427a-b93b-839dd480a585" containerID="5cf66975cb67c06b52c1326606e816a40a5cdb95959ea3e107cc911ef85699e9" exitCode=0 Nov 02 21:21:00.821352 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:00.821316 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g" Nov 02 21:21:00.821352 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:00.821325 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g" event={"ID":"dd46849e-e2c9-427a-b93b-839dd480a585","Type":"ContainerDied","Data":"5cf66975cb67c06b52c1326606e816a40a5cdb95959ea3e107cc911ef85699e9"} Nov 02 21:21:00.821832 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:00.821373 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g" event={"ID":"dd46849e-e2c9-427a-b93b-839dd480a585","Type":"ContainerDied","Data":"ee27e78cd938a657b8f0166b2a3fee71dcb670ba258d30cdd8ee3596941a646e"} Nov 02 21:21:00.821832 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:00.821395 4935 scope.go:122] "RemoveContainer" containerID="5cf66975cb67c06b52c1326606e816a40a5cdb95959ea3e107cc911ef85699e9" Nov 02 21:21:00.833567 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:00.833536 4935 scope.go:122] "RemoveContainer" containerID="5cf66975cb67c06b52c1326606e816a40a5cdb95959ea3e107cc911ef85699e9" Nov 02 21:21:00.833996 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:21:00.833975 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cf66975cb67c06b52c1326606e816a40a5cdb95959ea3e107cc911ef85699e9\": container with ID starting with 5cf66975cb67c06b52c1326606e816a40a5cdb95959ea3e107cc911ef85699e9 not found: ID does not exist" containerID="5cf66975cb67c06b52c1326606e816a40a5cdb95959ea3e107cc911ef85699e9" Nov 02 21:21:00.834041 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:00.834005 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cf66975cb67c06b52c1326606e816a40a5cdb95959ea3e107cc911ef85699e9"} err="failed to get container status \"5cf66975cb67c06b52c1326606e816a40a5cdb95959ea3e107cc911ef85699e9\": rpc error: code = NotFound desc = could not find container \"5cf66975cb67c06b52c1326606e816a40a5cdb95959ea3e107cc911ef85699e9\": container with ID starting with 5cf66975cb67c06b52c1326606e816a40a5cdb95959ea3e107cc911ef85699e9 not found: ID does not exist" Nov 02 21:21:00.973898 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:00.973862 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g"] Nov 02 21:21:01.020903 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:01.020873 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5478da31-7f6754dccc-fcf4g"] Nov 02 21:21:01.023121 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:01.023093 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj"] Nov 02 21:21:01.023352 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:01.023323 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj" podUID="51116ad1-b97f-42fa-80d2-a10cd0af3e45" containerName="maas-api" containerID="cri-o://9942ecaf6ab6ee398c698f27809d85e75d3fce8891388b5074cf6d1a7f961a06" gracePeriod=30 Nov 02 21:21:01.187528 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:01.187470 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:01.288672 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:01.288647 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj" Nov 02 21:21:01.358293 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:01.358263 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51116ad1-b97f-42fa-80d2-a10cd0af3e45-kube-api-access-2bpts\" (UniqueName: \"kubernetes.io/projected/51116ad1-b97f-42fa-80d2-a10cd0af3e45-kube-api-access-2bpts\") pod \"51116ad1-b97f-42fa-80d2-a10cd0af3e45\" (UID: \"51116ad1-b97f-42fa-80d2-a10cd0af3e45\") " Nov 02 21:21:01.358489 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:01.358308 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51116ad1-b97f-42fa-80d2-a10cd0af3e45-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51116ad1-b97f-42fa-80d2-a10cd0af3e45-maas-api-tls\") pod \"51116ad1-b97f-42fa-80d2-a10cd0af3e45\" (UID: \"51116ad1-b97f-42fa-80d2-a10cd0af3e45\") " Nov 02 21:21:01.358489 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:01.358371 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51116ad1-b97f-42fa-80d2-a10cd0af3e45-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51116ad1-b97f-42fa-80d2-a10cd0af3e45-openshift-service-ca\") pod \"51116ad1-b97f-42fa-80d2-a10cd0af3e45\" (UID: \"51116ad1-b97f-42fa-80d2-a10cd0af3e45\") " Nov 02 21:21:01.358489 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:01.358467 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51116ad1-b97f-42fa-80d2-a10cd0af3e45-metrics-certs\" (UniqueName: \"kubernetes.io/secret/51116ad1-b97f-42fa-80d2-a10cd0af3e45-metrics-certs\") pod \"51116ad1-b97f-42fa-80d2-a10cd0af3e45\" (UID: \"51116ad1-b97f-42fa-80d2-a10cd0af3e45\") " Nov 02 21:21:01.358753 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:01.358728 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51116ad1-b97f-42fa-80d2-a10cd0af3e45-openshift-service-ca" pod "51116ad1-b97f-42fa-80d2-a10cd0af3e45" (UID: "51116ad1-b97f-42fa-80d2-a10cd0af3e45"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:21:01.358895 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:01.358822 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51116ad1-b97f-42fa-80d2-a10cd0af3e45-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:01.360700 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:01.360667 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51116ad1-b97f-42fa-80d2-a10cd0af3e45-kube-api-access-2bpts" pod "51116ad1-b97f-42fa-80d2-a10cd0af3e45" (UID: "51116ad1-b97f-42fa-80d2-a10cd0af3e45"). InnerVolumeSpecName "kube-api-access-2bpts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:21:01.360780 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:01.360674 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51116ad1-b97f-42fa-80d2-a10cd0af3e45-metrics-certs" pod "51116ad1-b97f-42fa-80d2-a10cd0af3e45" (UID: "51116ad1-b97f-42fa-80d2-a10cd0af3e45"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:21:01.360780 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:01.360756 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51116ad1-b97f-42fa-80d2-a10cd0af3e45-maas-api-tls" pod "51116ad1-b97f-42fa-80d2-a10cd0af3e45" (UID: "51116ad1-b97f-42fa-80d2-a10cd0af3e45"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:21:01.459990 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:01.459903 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51116ad1-b97f-42fa-80d2-a10cd0af3e45-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:01.459990 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:01.459936 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bpts\" (UniqueName: \"kubernetes.io/projected/51116ad1-b97f-42fa-80d2-a10cd0af3e45-kube-api-access-2bpts\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:01.459990 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:01.459946 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51116ad1-b97f-42fa-80d2-a10cd0af3e45-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:01.775838 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:01.775804 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd46849e-e2c9-427a-b93b-839dd480a585" path="/var/lib/kubelet/pods/dd46849e-e2c9-427a-b93b-839dd480a585/volumes" Nov 02 21:21:01.827979 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:01.827948 4935 generic.go:356] "Generic (PLEG): container finished" podID="51116ad1-b97f-42fa-80d2-a10cd0af3e45" containerID="9942ecaf6ab6ee398c698f27809d85e75d3fce8891388b5074cf6d1a7f961a06" exitCode=0 Nov 02 21:21:01.828417 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:01.827984 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj" event={"ID":"51116ad1-b97f-42fa-80d2-a10cd0af3e45","Type":"ContainerDied","Data":"9942ecaf6ab6ee398c698f27809d85e75d3fce8891388b5074cf6d1a7f961a06"} Nov 02 21:21:01.828417 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:01.827999 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj" Nov 02 21:21:01.828417 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:01.828018 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj" event={"ID":"51116ad1-b97f-42fa-80d2-a10cd0af3e45","Type":"ContainerDied","Data":"965b52b78953cdb0c59d93e21edc538050d5ed06a5e14298aa37a984943d22ec"} Nov 02 21:21:01.828417 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:01.828040 4935 scope.go:122] "RemoveContainer" containerID="9942ecaf6ab6ee398c698f27809d85e75d3fce8891388b5074cf6d1a7f961a06" Nov 02 21:21:01.837276 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:01.837246 4935 scope.go:122] "RemoveContainer" containerID="9942ecaf6ab6ee398c698f27809d85e75d3fce8891388b5074cf6d1a7f961a06" Nov 02 21:21:01.837552 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:21:01.837533 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9942ecaf6ab6ee398c698f27809d85e75d3fce8891388b5074cf6d1a7f961a06\": container with ID starting with 9942ecaf6ab6ee398c698f27809d85e75d3fce8891388b5074cf6d1a7f961a06 not found: ID does not exist" containerID="9942ecaf6ab6ee398c698f27809d85e75d3fce8891388b5074cf6d1a7f961a06" Nov 02 21:21:01.837609 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:01.837560 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9942ecaf6ab6ee398c698f27809d85e75d3fce8891388b5074cf6d1a7f961a06"} err="failed to get container status \"9942ecaf6ab6ee398c698f27809d85e75d3fce8891388b5074cf6d1a7f961a06\": rpc error: code = NotFound desc = could not find container \"9942ecaf6ab6ee398c698f27809d85e75d3fce8891388b5074cf6d1a7f961a06\": container with ID starting with 9942ecaf6ab6ee398c698f27809d85e75d3fce8891388b5074cf6d1a7f961a06 not found: ID does not exist" Nov 02 21:21:01.843880 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:01.843858 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:01.980791 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:01.980756 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj"] Nov 02 21:21:02.026714 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:02.026642 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84de7f3a-5c96d69569-bgrvj"] Nov 02 21:21:02.109622 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:02.109573 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:02.362462 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:02.362045 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:02.586223 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:02.586185 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:02.904839 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:02.904800 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:03.154806 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:03.154727 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:03.371831 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:03.371792 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:03.691778 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:03.691685 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:03.775077 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:03.775041 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51116ad1-b97f-42fa-80d2-a10cd0af3e45" path="/var/lib/kubelet/pods/51116ad1-b97f-42fa-80d2-a10cd0af3e45/volumes" Nov 02 21:21:03.923196 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:03.923123 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:03.967150 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:03.967061 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg"] Nov 02 21:21:03.967496 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:03.967484 4935 cpu_manager.go:414] "containerMap: removing container" podUID="dd46849e-e2c9-427a-b93b-839dd480a585" containerName="maas-api" Nov 02 21:21:03.967544 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:03.967498 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd46849e-e2c9-427a-b93b-839dd480a585" containerName="maas-api" Nov 02 21:21:03.967544 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:03.967520 4935 cpu_manager.go:414] "containerMap: removing container" podUID="51116ad1-b97f-42fa-80d2-a10cd0af3e45" containerName="maas-api" Nov 02 21:21:03.967544 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:03.967525 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51116ad1-b97f-42fa-80d2-a10cd0af3e45" containerName="maas-api" Nov 02 21:21:03.967637 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:03.967596 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="51116ad1-b97f-42fa-80d2-a10cd0af3e45" containerName="maas-api" Nov 02 21:21:03.967637 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:03.967609 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd46849e-e2c9-427a-b93b-839dd480a585" containerName="maas-api" Nov 02 21:21:03.972453 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:03.972430 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.027335 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.027303 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg"] Nov 02 21:21:04.070453 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.070418 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-f51151-openshift-default-dockercfg-5nn46\"" Nov 02 21:21:04.084753 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.084722 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.084906 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.084756 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.084906 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.084782 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.084985 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.084938 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.084985 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.084974 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dae32e8-d70f-466c-8350-e1ae72e1372f-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.085075 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.085001 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dae32e8-d70f-466c-8350-e1ae72e1372f-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.085119 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.085069 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvnph\" (UniqueName: \"kubernetes.io/projected/9dae32e8-d70f-466c-8350-e1ae72e1372f-kube-api-access-bvnph\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.085192 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.085120 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dae32e8-d70f-466c-8350-e1ae72e1372f-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.085229 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.085198 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.158462 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.158419 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:04.186581 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.186548 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.186759 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.186588 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dae32e8-d70f-466c-8350-e1ae72e1372f-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.186759 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.186608 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dae32e8-d70f-466c-8350-e1ae72e1372f-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.186759 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.186630 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvnph\" (UniqueName: \"kubernetes.io/projected/9dae32e8-d70f-466c-8350-e1ae72e1372f-kube-api-access-bvnph\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.186759 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.186676 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dae32e8-d70f-466c-8350-e1ae72e1372f-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.186759 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.186717 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.187038 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.186760 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.187038 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.186783 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.187038 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.186819 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.187246 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.187095 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.187246 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.187168 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.187246 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.187219 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.187411 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.187384 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.187464 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.187421 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dae32e8-d70f-466c-8350-e1ae72e1372f-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.189208 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.189186 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.189433 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.189416 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dae32e8-d70f-466c-8350-e1ae72e1372f-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.240333 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.240265 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dae32e8-d70f-466c-8350-e1ae72e1372f-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.240450 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.240373 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvnph\" (UniqueName: \"kubernetes.io/projected/9dae32e8-d70f-466c-8350-e1ae72e1372f-kube-api-access-bvnph\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.288226 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.288196 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:04.380171 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.379486 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:04.619977 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.619941 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:04.703458 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.703425 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg"] Nov 02 21:21:04.707086 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:21:04.707020 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dae32e8_d70f_466c_8350_e1ae72e1372f.slice/crio-52a7ed7fe0a13ea6edfe4213bde18d91387668340c41443f4a9c41c3eca5bc60 WatchSource:0}: Error finding container 52a7ed7fe0a13ea6edfe4213bde18d91387668340c41443f4a9c41c3eca5bc60: Status 404 returned error can't find the container with id 52a7ed7fe0a13ea6edfe4213bde18d91387668340c41443f4a9c41c3eca5bc60 Nov 02 21:21:04.709450 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.709420 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:21:04.709542 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.709483 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:21:04.709542 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.709514 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:21:04.848597 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.848560 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" event={"ID":"9dae32e8-d70f-466c-8350-e1ae72e1372f","Type":"ContainerStarted","Data":"6ca7d14e9e01d80ac337890ce87c29db95be81f0082bd0914de1d08bb6d81fd4"} Nov 02 21:21:04.848743 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.848625 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" event={"ID":"9dae32e8-d70f-466c-8350-e1ae72e1372f","Type":"ContainerStarted","Data":"52a7ed7fe0a13ea6edfe4213bde18d91387668340c41443f4a9c41c3eca5bc60"} Nov 02 21:21:04.958859 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:04.958739 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" podStartSLOduration=1.95872066 podStartE2EDuration="1.95872066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:21:04.958250191 +0000 UTC m=+757.397824915" watchObservedRunningTime="2026-11-02 21:21:04.95872066 +0000 UTC m=+757.398295384" Nov 02 21:21:05.289310 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:05.289274 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:05.294844 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:05.294812 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:05.373673 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:05.373633 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:05.854837 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:05.854803 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:05.859153 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:05.856625 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:06.038272 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:06.038238 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:06.285922 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:06.285882 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:06.522044 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:06.522002 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:06.738484 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:06.738400 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:07.827040 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:07.826999 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:08.068766 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:08.068689 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:08.528658 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:08.528627 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:08.756244 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:08.756212 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:11.019293 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:11.019262 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:11.240684 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:11.240651 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:12.635362 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:12.635327 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:12.865671 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:12.865637 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:13.350421 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:13.350383 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j"] Nov 02 21:21:13.350643 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:13.350614 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j" podUID="deca8e64-4310-4e47-a1bb-b037781009c8" containerName="maas-api" containerID="cri-o://089a6a774a62f2a1547ea02b80a9714c92757d8bbe341e3754f56419bed2f529" gracePeriod=30 Nov 02 21:21:13.585981 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:13.578510 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:13.621574 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:13.621500 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j" Nov 02 21:21:13.780457 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:13.780404 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deca8e64-4310-4e47-a1bb-b037781009c8-kube-api-access-59wdc\" (UniqueName: \"kubernetes.io/projected/deca8e64-4310-4e47-a1bb-b037781009c8-kube-api-access-59wdc\") pod \"deca8e64-4310-4e47-a1bb-b037781009c8\" (UID: \"deca8e64-4310-4e47-a1bb-b037781009c8\") " Nov 02 21:21:13.786020 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:13.780546 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deca8e64-4310-4e47-a1bb-b037781009c8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/deca8e64-4310-4e47-a1bb-b037781009c8-metrics-certs\") pod \"deca8e64-4310-4e47-a1bb-b037781009c8\" (UID: \"deca8e64-4310-4e47-a1bb-b037781009c8\") " Nov 02 21:21:13.786020 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:13.780662 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deca8e64-4310-4e47-a1bb-b037781009c8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deca8e64-4310-4e47-a1bb-b037781009c8-openshift-service-ca\") pod \"deca8e64-4310-4e47-a1bb-b037781009c8\" (UID: \"deca8e64-4310-4e47-a1bb-b037781009c8\") " Nov 02 21:21:13.786020 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:13.780735 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deca8e64-4310-4e47-a1bb-b037781009c8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deca8e64-4310-4e47-a1bb-b037781009c8-maas-api-tls\") pod \"deca8e64-4310-4e47-a1bb-b037781009c8\" (UID: \"deca8e64-4310-4e47-a1bb-b037781009c8\") " Nov 02 21:21:13.786020 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:13.781049 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deca8e64-4310-4e47-a1bb-b037781009c8-openshift-service-ca" pod "deca8e64-4310-4e47-a1bb-b037781009c8" (UID: "deca8e64-4310-4e47-a1bb-b037781009c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:21:13.786020 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:13.782902 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deca8e64-4310-4e47-a1bb-b037781009c8-metrics-certs" pod "deca8e64-4310-4e47-a1bb-b037781009c8" (UID: "deca8e64-4310-4e47-a1bb-b037781009c8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:21:13.786020 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:13.783038 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deca8e64-4310-4e47-a1bb-b037781009c8-kube-api-access-59wdc" pod "deca8e64-4310-4e47-a1bb-b037781009c8" (UID: "deca8e64-4310-4e47-a1bb-b037781009c8"). InnerVolumeSpecName "kube-api-access-59wdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:21:13.786020 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:13.783200 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deca8e64-4310-4e47-a1bb-b037781009c8-maas-api-tls" pod "deca8e64-4310-4e47-a1bb-b037781009c8" (UID: "deca8e64-4310-4e47-a1bb-b037781009c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:21:13.881651 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:13.881577 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deca8e64-4310-4e47-a1bb-b037781009c8-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:13.881651 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:13.881611 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59wdc\" (UniqueName: \"kubernetes.io/projected/deca8e64-4310-4e47-a1bb-b037781009c8-kube-api-access-59wdc\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:13.881651 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:13.881633 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deca8e64-4310-4e47-a1bb-b037781009c8-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:13.881651 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:13.881648 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deca8e64-4310-4e47-a1bb-b037781009c8-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:13.900084 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:13.900055 4935 generic.go:356] "Generic (PLEG): container finished" podID="deca8e64-4310-4e47-a1bb-b037781009c8" containerID="089a6a774a62f2a1547ea02b80a9714c92757d8bbe341e3754f56419bed2f529" exitCode=0 Nov 02 21:21:13.900232 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:13.900116 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j" Nov 02 21:21:13.900232 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:13.900152 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j" event={"ID":"deca8e64-4310-4e47-a1bb-b037781009c8","Type":"ContainerDied","Data":"089a6a774a62f2a1547ea02b80a9714c92757d8bbe341e3754f56419bed2f529"} Nov 02 21:21:13.900232 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:13.900195 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j" event={"ID":"deca8e64-4310-4e47-a1bb-b037781009c8","Type":"ContainerDied","Data":"eb4c267d3cc1796526d465ffa8d1e6307aacd323091d196adacc0a78ce6fb54e"} Nov 02 21:21:13.900232 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:13.900216 4935 scope.go:122] "RemoveContainer" containerID="089a6a774a62f2a1547ea02b80a9714c92757d8bbe341e3754f56419bed2f529" Nov 02 21:21:13.909557 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:13.909541 4935 scope.go:122] "RemoveContainer" containerID="089a6a774a62f2a1547ea02b80a9714c92757d8bbe341e3754f56419bed2f529" Nov 02 21:21:13.909831 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:21:13.909807 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"089a6a774a62f2a1547ea02b80a9714c92757d8bbe341e3754f56419bed2f529\": container with ID starting with 089a6a774a62f2a1547ea02b80a9714c92757d8bbe341e3754f56419bed2f529 not found: ID does not exist" containerID="089a6a774a62f2a1547ea02b80a9714c92757d8bbe341e3754f56419bed2f529" Nov 02 21:21:13.909927 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:13.909839 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"089a6a774a62f2a1547ea02b80a9714c92757d8bbe341e3754f56419bed2f529"} err="failed to get container status \"089a6a774a62f2a1547ea02b80a9714c92757d8bbe341e3754f56419bed2f529\": rpc error: code = NotFound desc = could not find container \"089a6a774a62f2a1547ea02b80a9714c92757d8bbe341e3754f56419bed2f529\": container with ID starting with 089a6a774a62f2a1547ea02b80a9714c92757d8bbe341e3754f56419bed2f529 not found: ID does not exist" Nov 02 21:21:14.056510 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:14.056472 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j"] Nov 02 21:21:14.059723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:14.059685 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-54a47302-69594ff7f4-dzx5j"] Nov 02 21:21:14.124416 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:14.124378 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:14.351481 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:14.351432 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:14.994171 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:14.994113 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:15.598785 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:15.586910 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:15.779126 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:15.779095 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deca8e64-4310-4e47-a1bb-b037781009c8" path="/var/lib/kubelet/pods/deca8e64-4310-4e47-a1bb-b037781009c8/volumes" Nov 02 21:21:15.804045 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:15.804007 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:16.534731 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:16.534698 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl"] Nov 02 21:21:16.535119 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:16.535080 4935 cpu_manager.go:414] "containerMap: removing container" podUID="deca8e64-4310-4e47-a1bb-b037781009c8" containerName="maas-api" Nov 02 21:21:16.535119 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:16.535091 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deca8e64-4310-4e47-a1bb-b037781009c8" containerName="maas-api" Nov 02 21:21:16.535221 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:16.535199 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="deca8e64-4310-4e47-a1bb-b037781009c8" containerName="maas-api" Nov 02 21:21:16.539848 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:16.539827 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl" Nov 02 21:21:16.589668 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:16.589644 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-f51151-serving-cert\"" Nov 02 21:21:16.638392 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:16.638358 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl"] Nov 02 21:21:16.706965 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:16.706935 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49454e6d-962a-4cc9-8976-4fdb2442bfdb-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl\" (UID: \"49454e6d-962a-4cc9-8976-4fdb2442bfdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl" Nov 02 21:21:16.707124 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:16.707062 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49454e6d-962a-4cc9-8976-4fdb2442bfdb-maas-api-tls\") pod \"maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl\" (UID: \"49454e6d-962a-4cc9-8976-4fdb2442bfdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl" Nov 02 21:21:16.707124 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:16.707082 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzsdk\" (UniqueName: \"kubernetes.io/projected/49454e6d-962a-4cc9-8976-4fdb2442bfdb-kube-api-access-fzsdk\") pod \"maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl\" (UID: \"49454e6d-962a-4cc9-8976-4fdb2442bfdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl" Nov 02 21:21:16.707230 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:16.707207 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49454e6d-962a-4cc9-8976-4fdb2442bfdb-metrics-certs\") pod \"maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl\" (UID: \"49454e6d-962a-4cc9-8976-4fdb2442bfdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl" Nov 02 21:21:16.808501 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:16.808415 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49454e6d-962a-4cc9-8976-4fdb2442bfdb-maas-api-tls\") pod \"maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl\" (UID: \"49454e6d-962a-4cc9-8976-4fdb2442bfdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl" Nov 02 21:21:16.808501 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:16.808452 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzsdk\" (UniqueName: \"kubernetes.io/projected/49454e6d-962a-4cc9-8976-4fdb2442bfdb-kube-api-access-fzsdk\") pod \"maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl\" (UID: \"49454e6d-962a-4cc9-8976-4fdb2442bfdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl" Nov 02 21:21:16.808721 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:16.808528 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49454e6d-962a-4cc9-8976-4fdb2442bfdb-metrics-certs\") pod \"maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl\" (UID: \"49454e6d-962a-4cc9-8976-4fdb2442bfdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl" Nov 02 21:21:16.808721 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:16.808591 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49454e6d-962a-4cc9-8976-4fdb2442bfdb-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl\" (UID: \"49454e6d-962a-4cc9-8976-4fdb2442bfdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl" Nov 02 21:21:16.808721 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:21:16.808685 4935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-serving-cert: secret "maas-api-e2e-worker-w3-f51151-serving-cert" not found Nov 02 21:21:16.808885 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:21:16.808779 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49454e6d-962a-4cc9-8976-4fdb2442bfdb-maas-api-tls podName:49454e6d-962a-4cc9-8976-4fdb2442bfdb nodeName:}" failed. No retries permitted until 2026-11-02 21:21:17.308756161 +0000 UTC m=+769.748330866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/49454e6d-962a-4cc9-8976-4fdb2442bfdb-maas-api-tls") pod "maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl" (UID: "49454e6d-962a-4cc9-8976-4fdb2442bfdb") : secret "maas-api-e2e-worker-w3-f51151-serving-cert" not found Nov 02 21:21:16.809294 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:16.809272 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49454e6d-962a-4cc9-8976-4fdb2442bfdb-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl\" (UID: \"49454e6d-962a-4cc9-8976-4fdb2442bfdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl" Nov 02 21:21:16.811027 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:16.811005 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49454e6d-962a-4cc9-8976-4fdb2442bfdb-metrics-certs\") pod \"maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl\" (UID: \"49454e6d-962a-4cc9-8976-4fdb2442bfdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl" Nov 02 21:21:16.864306 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:16.864271 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzsdk\" (UniqueName: \"kubernetes.io/projected/49454e6d-962a-4cc9-8976-4fdb2442bfdb-kube-api-access-fzsdk\") pod \"maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl\" (UID: \"49454e6d-962a-4cc9-8976-4fdb2442bfdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl" Nov 02 21:21:16.864552 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:16.864516 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:17.081764 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:17.081654 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:17.307532 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:17.307501 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:17.312712 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:17.312683 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49454e6d-962a-4cc9-8976-4fdb2442bfdb-maas-api-tls\") pod \"maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl\" (UID: \"49454e6d-962a-4cc9-8976-4fdb2442bfdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl" Nov 02 21:21:17.315106 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:17.315082 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49454e6d-962a-4cc9-8976-4fdb2442bfdb-maas-api-tls\") pod \"maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl\" (UID: \"49454e6d-962a-4cc9-8976-4fdb2442bfdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl" Nov 02 21:21:17.452832 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:17.452738 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl" Nov 02 21:21:17.835200 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:17.834979 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl"] Nov 02 21:21:17.837852 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:21:17.837818 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49454e6d_962a_4cc9_8976_4fdb2442bfdb.slice/crio-9c68629593899c834d0b667e1dd8574271619718388d6878a129b1fd3c91693c WatchSource:0}: Error finding container 9c68629593899c834d0b667e1dd8574271619718388d6878a129b1fd3c91693c: Status 404 returned error can't find the container with id 9c68629593899c834d0b667e1dd8574271619718388d6878a129b1fd3c91693c Nov 02 21:21:17.861536 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:17.861508 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:17.922722 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:17.922674 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl" event={"ID":"49454e6d-962a-4cc9-8976-4fdb2442bfdb","Type":"ContainerStarted","Data":"9c68629593899c834d0b667e1dd8574271619718388d6878a129b1fd3c91693c"} Nov 02 21:21:18.133327 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:18.133250 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:18.471533 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:18.471438 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:18.695211 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:18.695163 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:19.336189 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:19.336090 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:19.932930 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:19.932898 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:19.942394 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:19.942364 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl" event={"ID":"49454e6d-962a-4cc9-8976-4fdb2442bfdb","Type":"ContainerStarted","Data":"7bb067fe72bcc216d7deaceb79e29a24a695e26ae05c9610277a0073242895b2"} Nov 02 21:21:19.942567 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:19.942507 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl" Nov 02 21:21:20.052418 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:20.052364 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl" podStartSLOduration=2.8247614 podStartE2EDuration="4.052350703s" totalImagesPullingTime="1.227589303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:21:20.050695669 +0000 UTC m=+772.490270391" watchObservedRunningTime="2026-11-02 21:21:20.052350703 +0000 UTC m=+772.491925425" Nov 02 21:21:20.213457 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:20.213364 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:21.035785 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:21.035743 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:21.368778 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:21.368675 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:21.693726 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:21.693629 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:21.923606 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:21.923572 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:22.495841 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:22.495799 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:22.741959 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:22.741924 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:23.006746 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:23.006666 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:23.232960 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:23.232923 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:23.458066 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:23.458032 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:24.876813 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:24.876777 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:25.110913 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:25.110878 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:25.954139 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:25.954110 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl" Nov 02 21:21:28.205529 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:28.205450 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:28.430328 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:28.430295 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:28.467571 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:21:28.467489 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:21:29.999810 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:29.999768 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:30.227540 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:30.227509 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:30.581405 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:30.581370 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:30.814593 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:30.814549 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:33.812611 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:33.812578 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:39.356276 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:39.356233 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:39.576377 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:39.576310 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:41.094476 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:41.094440 4935 generic.go:356] "Generic (PLEG): container finished" podID="9dae32e8-d70f-466c-8350-e1ae72e1372f" containerID="6ca7d14e9e01d80ac337890ce87c29db95be81f0082bd0914de1d08bb6d81fd4" exitCode=0 Nov 02 21:21:41.095038 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:41.094519 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" event={"ID":"9dae32e8-d70f-466c-8350-e1ae72e1372f","Type":"ContainerDied","Data":"6ca7d14e9e01d80ac337890ce87c29db95be81f0082bd0914de1d08bb6d81fd4"} Nov 02 21:21:41.095113 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:41.095036 4935 scope.go:122] "RemoveContainer" containerID="6ca7d14e9e01d80ac337890ce87c29db95be81f0082bd0914de1d08bb6d81fd4" Nov 02 21:21:41.096351 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:41.096323 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:21:41.096445 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:41.096388 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:21:41.096445 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:41.096421 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:21:42.101668 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:42.101633 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" event={"ID":"9dae32e8-d70f-466c-8350-e1ae72e1372f","Type":"ContainerStarted","Data":"77ac75a5e3a1bc1f25b55ff07f9d183d89faf3ea7e471fc73312d2ad2c9f4728"} Nov 02 21:21:42.288631 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:42.288598 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:42.288831 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:42.288804 4935 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 02 21:21:42.288908 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:42.288862 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" podUID="9dae32e8-d70f-466c-8350-e1ae72e1372f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 02 21:21:43.289017 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:43.288981 4935 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 02 21:21:43.289423 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:43.289027 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" podUID="9dae32e8-d70f-466c-8350-e1ae72e1372f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 02 21:21:44.288591 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:44.288553 4935 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 02 21:21:44.288773 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:44.288602 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" podUID="9dae32e8-d70f-466c-8350-e1ae72e1372f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 02 21:21:44.572638 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:44.572551 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:45.289040 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:45.289004 4935 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 02 21:21:45.289238 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:45.289054 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" podUID="9dae32e8-d70f-466c-8350-e1ae72e1372f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 02 21:21:46.289144 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:46.289092 4935 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 02 21:21:46.289630 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:46.289160 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" podUID="9dae32e8-d70f-466c-8350-e1ae72e1372f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 02 21:21:47.382264 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:47.382228 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:47.382843 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:47.382829 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:47.383900 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:47.383874 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 21:21:47.993152 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:47.993104 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:48.582283 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:48.582250 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:48.811707 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:48.811678 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:49.687426 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:49.687393 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws"] Nov 02 21:21:49.690180 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:49.690159 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws" Nov 02 21:21:49.740796 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:49.740766 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hrr7\" (UniqueName: \"kubernetes.io/projected/84fe42ff-d1d9-4f93-82a3-418ce33034df-kube-api-access-2hrr7\") pod \"maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws\" (UID: \"84fe42ff-d1d9-4f93-82a3-418ce33034df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws" Nov 02 21:21:49.740940 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:49.740817 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84fe42ff-d1d9-4f93-82a3-418ce33034df-metrics-certs\") pod \"maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws\" (UID: \"84fe42ff-d1d9-4f93-82a3-418ce33034df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws" Nov 02 21:21:49.740940 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:49.740904 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fe42ff-d1d9-4f93-82a3-418ce33034df-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws\" (UID: \"84fe42ff-d1d9-4f93-82a3-418ce33034df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws" Nov 02 21:21:49.741056 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:49.740948 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84fe42ff-d1d9-4f93-82a3-418ce33034df-maas-api-tls\") pod \"maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws\" (UID: \"84fe42ff-d1d9-4f93-82a3-418ce33034df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws" Nov 02 21:21:49.787180 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:49.787148 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-6fa533-serving-cert\"" Nov 02 21:21:49.789636 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:49.789613 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws"] Nov 02 21:21:49.842475 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:49.842444 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hrr7\" (UniqueName: \"kubernetes.io/projected/84fe42ff-d1d9-4f93-82a3-418ce33034df-kube-api-access-2hrr7\") pod \"maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws\" (UID: \"84fe42ff-d1d9-4f93-82a3-418ce33034df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws" Nov 02 21:21:49.842645 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:49.842487 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84fe42ff-d1d9-4f93-82a3-418ce33034df-metrics-certs\") pod \"maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws\" (UID: \"84fe42ff-d1d9-4f93-82a3-418ce33034df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws" Nov 02 21:21:49.842645 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:49.842572 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fe42ff-d1d9-4f93-82a3-418ce33034df-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws\" (UID: \"84fe42ff-d1d9-4f93-82a3-418ce33034df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws" Nov 02 21:21:49.842645 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:49.842619 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84fe42ff-d1d9-4f93-82a3-418ce33034df-maas-api-tls\") pod \"maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws\" (UID: \"84fe42ff-d1d9-4f93-82a3-418ce33034df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws" Nov 02 21:21:49.842817 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:21:49.842722 4935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-serving-cert: secret "maas-api-e2e-worker-w2-6fa533-serving-cert" not found Nov 02 21:21:49.842817 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:21:49.842785 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84fe42ff-d1d9-4f93-82a3-418ce33034df-maas-api-tls podName:84fe42ff-d1d9-4f93-82a3-418ce33034df nodeName:}" failed. No retries permitted until 2026-11-02 21:21:50.342765686 +0000 UTC m=+802.782340400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/84fe42ff-d1d9-4f93-82a3-418ce33034df-maas-api-tls") pod "maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws" (UID: "84fe42ff-d1d9-4f93-82a3-418ce33034df") : secret "maas-api-e2e-worker-w2-6fa533-serving-cert" not found Nov 02 21:21:49.843355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:49.843328 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fe42ff-d1d9-4f93-82a3-418ce33034df-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws\" (UID: \"84fe42ff-d1d9-4f93-82a3-418ce33034df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws" Nov 02 21:21:49.845018 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:49.844999 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84fe42ff-d1d9-4f93-82a3-418ce33034df-metrics-certs\") pod \"maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws\" (UID: \"84fe42ff-d1d9-4f93-82a3-418ce33034df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws" Nov 02 21:21:49.896194 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:49.896160 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hrr7\" (UniqueName: \"kubernetes.io/projected/84fe42ff-d1d9-4f93-82a3-418ce33034df-kube-api-access-2hrr7\") pod \"maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws\" (UID: \"84fe42ff-d1d9-4f93-82a3-418ce33034df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws" Nov 02 21:21:50.074071 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:50.074038 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:50.301517 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:50.301486 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:50.347907 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:50.347831 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84fe42ff-d1d9-4f93-82a3-418ce33034df-maas-api-tls\") pod \"maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws\" (UID: \"84fe42ff-d1d9-4f93-82a3-418ce33034df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws" Nov 02 21:21:50.350346 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:50.350325 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84fe42ff-d1d9-4f93-82a3-418ce33034df-maas-api-tls\") pod \"maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws\" (UID: \"84fe42ff-d1d9-4f93-82a3-418ce33034df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws" Nov 02 21:21:50.602233 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:50.602159 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws" Nov 02 21:21:50.966358 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:50.966334 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws"] Nov 02 21:21:50.968469 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:21:50.968441 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84fe42ff_d1d9_4f93_82a3_418ce33034df.slice/crio-02131d28618671602bf817e015e8516f9c5cce0313cd0f44b00461c8e6d0ab48 WatchSource:0}: Error finding container 02131d28618671602bf817e015e8516f9c5cce0313cd0f44b00461c8e6d0ab48: Status 404 returned error can't find the container with id 02131d28618671602bf817e015e8516f9c5cce0313cd0f44b00461c8e6d0ab48 Nov 02 21:21:51.152333 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:51.152296 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws" event={"ID":"84fe42ff-d1d9-4f93-82a3-418ce33034df","Type":"ContainerStarted","Data":"02131d28618671602bf817e015e8516f9c5cce0313cd0f44b00461c8e6d0ab48"} Nov 02 21:21:51.333199 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:51.333170 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s"] Nov 02 21:21:51.336995 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:51.336969 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" Nov 02 21:21:51.389766 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:51.389720 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f51151\"/\"kube-root-ca.crt\"" Nov 02 21:21:51.432854 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:51.432815 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f51151\"/\"simulator-w3-f51151-kserve-self-signed-certs\"" Nov 02 21:21:51.432999 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:51.432936 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f51151\"/\"openshift-service-ca.crt\"" Nov 02 21:21:51.432999 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:51.432946 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f51151\"/\"default-dockercfg-2ss8g\"" Nov 02 21:21:51.436771 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:51.436748 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s"] Nov 02 21:21:51.457807 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:51.457761 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-tls-certs\") pod \"simulator-w3-f51151-kserve-869b5bf9bf-8jk2s\" (UID: \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\") " pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" Nov 02 21:21:51.458073 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:51.458034 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-dshm\") pod \"simulator-w3-f51151-kserve-869b5bf9bf-8jk2s\" (UID: \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\") " pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" Nov 02 21:21:51.458169 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:51.458145 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p7t8\" (UniqueName: \"kubernetes.io/projected/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-kube-api-access-2p7t8\") pod \"simulator-w3-f51151-kserve-869b5bf9bf-8jk2s\" (UID: \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\") " pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" Nov 02 21:21:51.458264 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:51.458244 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-home\") pod \"simulator-w3-f51151-kserve-869b5bf9bf-8jk2s\" (UID: \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\") " pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" Nov 02 21:21:51.458325 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:51.458277 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-tmp-dir\") pod \"simulator-w3-f51151-kserve-869b5bf9bf-8jk2s\" (UID: \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\") " pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" Nov 02 21:21:51.458325 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:51.458311 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-model-cache\") pod \"simulator-w3-f51151-kserve-869b5bf9bf-8jk2s\" (UID: \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\") " pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" Nov 02 21:21:51.498560 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:51.498518 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:51.559796 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:51.559764 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-dshm\") pod \"simulator-w3-f51151-kserve-869b5bf9bf-8jk2s\" (UID: \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\") " pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" Nov 02 21:21:51.559961 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:51.559809 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p7t8\" (UniqueName: \"kubernetes.io/projected/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-kube-api-access-2p7t8\") pod \"simulator-w3-f51151-kserve-869b5bf9bf-8jk2s\" (UID: \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\") " pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" Nov 02 21:21:51.559961 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:51.559832 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-home\") pod \"simulator-w3-f51151-kserve-869b5bf9bf-8jk2s\" (UID: \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\") " pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" Nov 02 21:21:51.559961 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:51.559850 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-tmp-dir\") pod \"simulator-w3-f51151-kserve-869b5bf9bf-8jk2s\" (UID: \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\") " pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" Nov 02 21:21:51.559961 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:51.559876 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-model-cache\") pod \"simulator-w3-f51151-kserve-869b5bf9bf-8jk2s\" (UID: \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\") " pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" Nov 02 21:21:51.559961 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:51.559943 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-tls-certs\") pod \"simulator-w3-f51151-kserve-869b5bf9bf-8jk2s\" (UID: \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\") " pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" Nov 02 21:21:51.560360 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:51.560329 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-tmp-dir\") pod \"simulator-w3-f51151-kserve-869b5bf9bf-8jk2s\" (UID: \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\") " pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" Nov 02 21:21:51.560360 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:51.560349 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-home\") pod \"simulator-w3-f51151-kserve-869b5bf9bf-8jk2s\" (UID: \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\") " pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" Nov 02 21:21:51.560524 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:51.560396 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-model-cache\") pod \"simulator-w3-f51151-kserve-869b5bf9bf-8jk2s\" (UID: \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\") " pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" Nov 02 21:21:51.562030 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:51.562009 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-dshm\") pod \"simulator-w3-f51151-kserve-869b5bf9bf-8jk2s\" (UID: \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\") " pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" Nov 02 21:21:51.562309 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:51.562292 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-tls-certs\") pod \"simulator-w3-f51151-kserve-869b5bf9bf-8jk2s\" (UID: \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\") " pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" Nov 02 21:21:51.613040 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:51.612961 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p7t8\" (UniqueName: \"kubernetes.io/projected/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-kube-api-access-2p7t8\") pod \"simulator-w3-f51151-kserve-869b5bf9bf-8jk2s\" (UID: \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\") " pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" Nov 02 21:21:51.658875 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:51.658845 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" Nov 02 21:21:52.027765 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:52.027717 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s"] Nov 02 21:21:52.159613 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:52.159526 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws" event={"ID":"84fe42ff-d1d9-4f93-82a3-418ce33034df","Type":"ContainerStarted","Data":"c9d468cdd8638d3c8dba604a3e7a33845cbde12333864b917e096187d37dd46d"} Nov 02 21:21:52.159777 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:52.159643 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws" Nov 02 21:21:52.160903 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:52.160874 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" event={"ID":"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8","Type":"ContainerStarted","Data":"e77a2a43cf615dd52a37bf2b7347d39e5702b1d05a1fab4deae14ddb5633255f"} Nov 02 21:21:52.268673 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:52.268619 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws" podStartSLOduration=2.432794065 podStartE2EDuration="3.268605046s" totalImagesPullingTime="835.810981ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:21:52.266880822 +0000 UTC m=+804.706455544" watchObservedRunningTime="2026-11-02 21:21:52.268605046 +0000 UTC m=+804.708179768" Nov 02 21:21:53.401874 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:53.401831 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:53.985632 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:53.985595 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:54.213489 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:54.213399 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:54.842922 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:54.842871 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:55.181200 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:55.181101 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" event={"ID":"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8","Type":"ContainerStarted","Data":"292db166abf313cb5281582a076f2403da95de31756a916f594c51b2cb90b16d"} Nov 02 21:21:55.288418 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:55.288364 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" podStartSLOduration=2.207425303 podStartE2EDuration="4.28834954s" totalImagesPullingTime="2.080924237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:21:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:21:55.287564768 +0000 UTC m=+807.727139490" watchObservedRunningTime="2026-11-02 21:21:55.28834954 +0000 UTC m=+807.727924261" Nov 02 21:21:55.488779 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:55.488658 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:55.659842 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:55.659808 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" Nov 02 21:21:55.766639 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:55.766605 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:21:58.172581 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:21:58.172551 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws" Nov 02 21:21:58.471525 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:21:58.471428 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:22:00.732275 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:00.732237 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:01.659761 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:01.659721 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" Nov 02 21:22:01.672865 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:01.672837 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" Nov 02 21:22:02.227637 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:02.227608 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" Nov 02 21:22:05.746097 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:05.746057 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:10.763249 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:10.763213 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:11.186780 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:11.186667 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:11.463750 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:11.463646 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:11.710549 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:11.710509 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:14.975258 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:14.975219 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm"] Nov 02 21:22:14.978702 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:14.978678 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" Nov 02 21:22:15.077670 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:15.077632 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d6cb38\"/\"openshift-service-ca.crt\"" Nov 02 21:22:15.077876 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:15.077672 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d6cb38\"/\"default-dockercfg-rtxrz\"" Nov 02 21:22:15.077876 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:15.077743 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d6cb38\"/\"simulator-w4-d6cb38-kserve-self-signed-certs\"" Nov 02 21:22:15.077876 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:15.077792 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d6cb38\"/\"kube-root-ca.crt\"" Nov 02 21:22:15.080150 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:15.080111 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm"] Nov 02 21:22:15.096399 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:15.096372 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3920de75-9c3e-4826-8de5-22fcc6d50b0e-tls-certs\") pod \"simulator-w4-d6cb38-kserve-7588b6879f-4mmqm\" (UID: \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\") " pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" Nov 02 21:22:15.096496 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:15.096417 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85rrw\" (UniqueName: \"kubernetes.io/projected/3920de75-9c3e-4826-8de5-22fcc6d50b0e-kube-api-access-85rrw\") pod \"simulator-w4-d6cb38-kserve-7588b6879f-4mmqm\" (UID: \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\") " pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" Nov 02 21:22:15.096549 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:15.096504 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3920de75-9c3e-4826-8de5-22fcc6d50b0e-dshm\") pod \"simulator-w4-d6cb38-kserve-7588b6879f-4mmqm\" (UID: \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\") " pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" Nov 02 21:22:15.096549 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:15.096530 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3920de75-9c3e-4826-8de5-22fcc6d50b0e-tmp-dir\") pod \"simulator-w4-d6cb38-kserve-7588b6879f-4mmqm\" (UID: \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\") " pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" Nov 02 21:22:15.096649 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:15.096625 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3920de75-9c3e-4826-8de5-22fcc6d50b0e-model-cache\") pod \"simulator-w4-d6cb38-kserve-7588b6879f-4mmqm\" (UID: \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\") " pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" Nov 02 21:22:15.096707 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:15.096693 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3920de75-9c3e-4826-8de5-22fcc6d50b0e-home\") pod \"simulator-w4-d6cb38-kserve-7588b6879f-4mmqm\" (UID: \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\") " pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" Nov 02 21:22:15.141933 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:15.141897 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:15.198000 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:15.197965 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3920de75-9c3e-4826-8de5-22fcc6d50b0e-home\") pod \"simulator-w4-d6cb38-kserve-7588b6879f-4mmqm\" (UID: \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\") " pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" Nov 02 21:22:15.198179 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:15.198010 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3920de75-9c3e-4826-8de5-22fcc6d50b0e-tls-certs\") pod \"simulator-w4-d6cb38-kserve-7588b6879f-4mmqm\" (UID: \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\") " pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" Nov 02 21:22:15.198179 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:15.198054 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85rrw\" (UniqueName: \"kubernetes.io/projected/3920de75-9c3e-4826-8de5-22fcc6d50b0e-kube-api-access-85rrw\") pod \"simulator-w4-d6cb38-kserve-7588b6879f-4mmqm\" (UID: \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\") " pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" Nov 02 21:22:15.198179 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:15.198080 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3920de75-9c3e-4826-8de5-22fcc6d50b0e-dshm\") pod \"simulator-w4-d6cb38-kserve-7588b6879f-4mmqm\" (UID: \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\") " pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" Nov 02 21:22:15.198179 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:15.198099 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3920de75-9c3e-4826-8de5-22fcc6d50b0e-tmp-dir\") pod \"simulator-w4-d6cb38-kserve-7588b6879f-4mmqm\" (UID: \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\") " pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" Nov 02 21:22:15.198179 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:15.198172 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3920de75-9c3e-4826-8de5-22fcc6d50b0e-model-cache\") pod \"simulator-w4-d6cb38-kserve-7588b6879f-4mmqm\" (UID: \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\") " pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" Nov 02 21:22:15.198528 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:15.198494 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3920de75-9c3e-4826-8de5-22fcc6d50b0e-home\") pod \"simulator-w4-d6cb38-kserve-7588b6879f-4mmqm\" (UID: \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\") " pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" Nov 02 21:22:15.198599 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:15.198557 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3920de75-9c3e-4826-8de5-22fcc6d50b0e-tmp-dir\") pod \"simulator-w4-d6cb38-kserve-7588b6879f-4mmqm\" (UID: \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\") " pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" Nov 02 21:22:15.198730 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:15.198710 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3920de75-9c3e-4826-8de5-22fcc6d50b0e-model-cache\") pod \"simulator-w4-d6cb38-kserve-7588b6879f-4mmqm\" (UID: \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\") " pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" Nov 02 21:22:15.200475 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:15.200449 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3920de75-9c3e-4826-8de5-22fcc6d50b0e-dshm\") pod \"simulator-w4-d6cb38-kserve-7588b6879f-4mmqm\" (UID: \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\") " pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" Nov 02 21:22:15.200743 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:15.200724 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3920de75-9c3e-4826-8de5-22fcc6d50b0e-tls-certs\") pod \"simulator-w4-d6cb38-kserve-7588b6879f-4mmqm\" (UID: \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\") " pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" Nov 02 21:22:15.251871 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:15.251783 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85rrw\" (UniqueName: \"kubernetes.io/projected/3920de75-9c3e-4826-8de5-22fcc6d50b0e-kube-api-access-85rrw\") pod \"simulator-w4-d6cb38-kserve-7588b6879f-4mmqm\" (UID: \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\") " pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" Nov 02 21:22:15.294179 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:15.294121 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" Nov 02 21:22:15.668611 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:15.668585 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm"] Nov 02 21:22:15.670762 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:22:15.670736 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3920de75_9c3e_4826_8de5_22fcc6d50b0e.slice/crio-00fff1e9520b13deb9a72a0024758972063edb8a39da2a2ccda0098accd25014 WatchSource:0}: Error finding container 00fff1e9520b13deb9a72a0024758972063edb8a39da2a2ccda0098accd25014: Status 404 returned error can't find the container with id 00fff1e9520b13deb9a72a0024758972063edb8a39da2a2ccda0098accd25014 Nov 02 21:22:16.253439 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:16.253388 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:16.300903 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:16.300863 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" event={"ID":"3920de75-9c3e-4826-8de5-22fcc6d50b0e","Type":"ContainerStarted","Data":"fb046d23c6efc411fd4cf323c6be57faf02a543f14805601fd86e5e409bbc374"} Nov 02 21:22:16.301059 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:16.300922 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" event={"ID":"3920de75-9c3e-4826-8de5-22fcc6d50b0e","Type":"ContainerStarted","Data":"00fff1e9520b13deb9a72a0024758972063edb8a39da2a2ccda0098accd25014"} Nov 02 21:22:16.408926 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:16.408870 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" podStartSLOduration=2.408854897 podStartE2EDuration="2.408854897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:22:16.407908673 +0000 UTC m=+828.847483395" watchObservedRunningTime="2026-11-02 21:22:16.408854897 +0000 UTC m=+828.848429629" Nov 02 21:22:17.296531 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:17.296500 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" Nov 02 21:22:20.972030 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:20.971994 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f"] Nov 02 21:22:20.976060 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:20.976040 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" Nov 02 21:22:21.026061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:21.026001 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0c9014\"/\"kube-root-ca.crt\"" Nov 02 21:22:21.063384 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:21.063350 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkbvg\" (UniqueName: \"kubernetes.io/projected/aab5db02-4890-45e1-bcf8-1c43c335a393-kube-api-access-qkbvg\") pod \"simulator-w0-0c9014-kserve-6f66c899c6-59v7f\" (UID: \"aab5db02-4890-45e1-bcf8-1c43c335a393\") " pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" Nov 02 21:22:21.063570 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:21.063506 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aab5db02-4890-45e1-bcf8-1c43c335a393-tls-certs\") pod \"simulator-w0-0c9014-kserve-6f66c899c6-59v7f\" (UID: \"aab5db02-4890-45e1-bcf8-1c43c335a393\") " pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" Nov 02 21:22:21.063641 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:21.063606 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aab5db02-4890-45e1-bcf8-1c43c335a393-home\") pod \"simulator-w0-0c9014-kserve-6f66c899c6-59v7f\" (UID: \"aab5db02-4890-45e1-bcf8-1c43c335a393\") " pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" Nov 02 21:22:21.063705 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:21.063668 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aab5db02-4890-45e1-bcf8-1c43c335a393-tmp-dir\") pod \"simulator-w0-0c9014-kserve-6f66c899c6-59v7f\" (UID: \"aab5db02-4890-45e1-bcf8-1c43c335a393\") " pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" Nov 02 21:22:21.063760 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:21.063718 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aab5db02-4890-45e1-bcf8-1c43c335a393-dshm\") pod \"simulator-w0-0c9014-kserve-6f66c899c6-59v7f\" (UID: \"aab5db02-4890-45e1-bcf8-1c43c335a393\") " pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" Nov 02 21:22:21.063818 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:21.063760 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aab5db02-4890-45e1-bcf8-1c43c335a393-model-cache\") pod \"simulator-w0-0c9014-kserve-6f66c899c6-59v7f\" (UID: \"aab5db02-4890-45e1-bcf8-1c43c335a393\") " pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" Nov 02 21:22:21.072266 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:21.072240 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0c9014\"/\"default-dockercfg-c9nbm\"" Nov 02 21:22:21.072399 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:21.072364 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0c9014\"/\"simulator-w0-0c9014-kserve-self-signed-certs\"" Nov 02 21:22:21.072399 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:21.072388 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0c9014\"/\"openshift-service-ca.crt\"" Nov 02 21:22:21.074758 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:21.074739 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f"] Nov 02 21:22:21.159038 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:21.159004 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:21.164264 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:21.164235 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aab5db02-4890-45e1-bcf8-1c43c335a393-home\") pod \"simulator-w0-0c9014-kserve-6f66c899c6-59v7f\" (UID: \"aab5db02-4890-45e1-bcf8-1c43c335a393\") " pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" Nov 02 21:22:21.164403 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:21.164284 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aab5db02-4890-45e1-bcf8-1c43c335a393-tmp-dir\") pod \"simulator-w0-0c9014-kserve-6f66c899c6-59v7f\" (UID: \"aab5db02-4890-45e1-bcf8-1c43c335a393\") " pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" Nov 02 21:22:21.164403 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:21.164314 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aab5db02-4890-45e1-bcf8-1c43c335a393-dshm\") pod \"simulator-w0-0c9014-kserve-6f66c899c6-59v7f\" (UID: \"aab5db02-4890-45e1-bcf8-1c43c335a393\") " pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" Nov 02 21:22:21.164403 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:21.164353 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aab5db02-4890-45e1-bcf8-1c43c335a393-model-cache\") pod \"simulator-w0-0c9014-kserve-6f66c899c6-59v7f\" (UID: \"aab5db02-4890-45e1-bcf8-1c43c335a393\") " pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" Nov 02 21:22:21.164403 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:21.164381 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkbvg\" (UniqueName: \"kubernetes.io/projected/aab5db02-4890-45e1-bcf8-1c43c335a393-kube-api-access-qkbvg\") pod \"simulator-w0-0c9014-kserve-6f66c899c6-59v7f\" (UID: \"aab5db02-4890-45e1-bcf8-1c43c335a393\") " pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" Nov 02 21:22:21.164616 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:21.164499 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aab5db02-4890-45e1-bcf8-1c43c335a393-tls-certs\") pod \"simulator-w0-0c9014-kserve-6f66c899c6-59v7f\" (UID: \"aab5db02-4890-45e1-bcf8-1c43c335a393\") " pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" Nov 02 21:22:21.164673 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:21.164658 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aab5db02-4890-45e1-bcf8-1c43c335a393-home\") pod \"simulator-w0-0c9014-kserve-6f66c899c6-59v7f\" (UID: \"aab5db02-4890-45e1-bcf8-1c43c335a393\") " pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" Nov 02 21:22:21.164731 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:21.164685 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aab5db02-4890-45e1-bcf8-1c43c335a393-tmp-dir\") pod \"simulator-w0-0c9014-kserve-6f66c899c6-59v7f\" (UID: \"aab5db02-4890-45e1-bcf8-1c43c335a393\") " pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" Nov 02 21:22:21.164805 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:21.164784 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aab5db02-4890-45e1-bcf8-1c43c335a393-model-cache\") pod \"simulator-w0-0c9014-kserve-6f66c899c6-59v7f\" (UID: \"aab5db02-4890-45e1-bcf8-1c43c335a393\") " pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" Nov 02 21:22:21.166638 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:21.166611 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aab5db02-4890-45e1-bcf8-1c43c335a393-dshm\") pod \"simulator-w0-0c9014-kserve-6f66c899c6-59v7f\" (UID: \"aab5db02-4890-45e1-bcf8-1c43c335a393\") " pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" Nov 02 21:22:21.167211 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:21.167190 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aab5db02-4890-45e1-bcf8-1c43c335a393-tls-certs\") pod \"simulator-w0-0c9014-kserve-6f66c899c6-59v7f\" (UID: \"aab5db02-4890-45e1-bcf8-1c43c335a393\") " pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" Nov 02 21:22:21.217885 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:21.217851 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkbvg\" (UniqueName: \"kubernetes.io/projected/aab5db02-4890-45e1-bcf8-1c43c335a393-kube-api-access-qkbvg\") pod \"simulator-w0-0c9014-kserve-6f66c899c6-59v7f\" (UID: \"aab5db02-4890-45e1-bcf8-1c43c335a393\") " pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" Nov 02 21:22:21.288766 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:21.288737 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" Nov 02 21:22:21.370241 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:21.370204 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:21.663076 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:21.663049 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f"] Nov 02 21:22:21.665549 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:22:21.665511 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaab5db02_4890_45e1_bcf8_1c43c335a393.slice/crio-46bedece67c5c6cd1f61abed6a62e7142fca61602118adbb50772bfd273144e9 WatchSource:0}: Error finding container 46bedece67c5c6cd1f61abed6a62e7142fca61602118adbb50772bfd273144e9: Status 404 returned error can't find the container with id 46bedece67c5c6cd1f61abed6a62e7142fca61602118adbb50772bfd273144e9 Nov 02 21:22:22.006431 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:22.006342 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:22.336220 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:22.336175 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" event={"ID":"aab5db02-4890-45e1-bcf8-1c43c335a393","Type":"ContainerStarted","Data":"d5c8f4499b8bfe4c3107ad9bda7019e0afd3bf3d1a4f0b2e65d80c138de16804"} Nov 02 21:22:22.336398 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:22.336232 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" event={"ID":"aab5db02-4890-45e1-bcf8-1c43c335a393","Type":"ContainerStarted","Data":"46bedece67c5c6cd1f61abed6a62e7142fca61602118adbb50772bfd273144e9"} Nov 02 21:22:22.444859 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:22.444804 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" podStartSLOduration=2.444789677 podStartE2EDuration="2.444789677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:22:22.441931856 +0000 UTC m=+834.881506579" watchObservedRunningTime="2026-11-02 21:22:22.444789677 +0000 UTC m=+834.884364399" Nov 02 21:22:23.288888 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:23.288859 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" Nov 02 21:22:25.296912 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:25.296870 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" Nov 02 21:22:25.309569 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:25.309543 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" Nov 02 21:22:25.363547 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:25.363519 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" Nov 02 21:22:26.361605 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:26.361565 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:28.469061 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:22:28.469015 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:22:30.038187 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:30.038123 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:30.694206 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:30.694167 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:30.919406 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:30.919373 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:31.289866 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:31.289834 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" Nov 02 21:22:31.302445 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:31.302416 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" Nov 02 21:22:31.403052 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:31.403025 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" Nov 02 21:22:31.781123 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:31.781087 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:33.207091 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:33.207054 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:33.747119 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:33.747083 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:34.026426 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:34.026389 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:37.205614 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:37.205571 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:37.788932 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:37.788896 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:42.311043 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:42.311009 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:42.821354 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:42.821321 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:47.445195 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:47.445155 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:51.127864 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:51.127816 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9"] Nov 02 21:22:51.132913 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:51.132876 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" Nov 02 21:22:51.184873 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:51.184837 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9"] Nov 02 21:22:51.226808 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:51.226778 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-6fa533\"/\"openshift-service-ca.crt\"" Nov 02 21:22:51.226995 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:51.226814 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-6fa533\"/\"kube-root-ca.crt\"" Nov 02 21:22:51.228341 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:51.228322 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6fa533\"/\"simulator-w2-6fa533-kserve-self-signed-certs\"" Nov 02 21:22:51.228463 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:51.228454 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6fa533\"/\"default-dockercfg-lt567\"" Nov 02 21:22:51.281961 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:51.281919 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-home\") pod \"simulator-w2-6fa533-kserve-658676577-55vz9\" (UID: \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\") " pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" Nov 02 21:22:51.282170 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:51.281981 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-tmp-dir\") pod \"simulator-w2-6fa533-kserve-658676577-55vz9\" (UID: \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\") " pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" Nov 02 21:22:51.282170 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:51.282059 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2knmw\" (UniqueName: \"kubernetes.io/projected/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-kube-api-access-2knmw\") pod \"simulator-w2-6fa533-kserve-658676577-55vz9\" (UID: \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\") " pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" Nov 02 21:22:51.282170 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:51.282097 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-model-cache\") pod \"simulator-w2-6fa533-kserve-658676577-55vz9\" (UID: \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\") " pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" Nov 02 21:22:51.282170 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:51.282121 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-tls-certs\") pod \"simulator-w2-6fa533-kserve-658676577-55vz9\" (UID: \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\") " pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" Nov 02 21:22:51.282332 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:51.282284 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-dshm\") pod \"simulator-w2-6fa533-kserve-658676577-55vz9\" (UID: \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\") " pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" Nov 02 21:22:51.344103 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:51.344056 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:51.383192 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:51.383094 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-home\") pod \"simulator-w2-6fa533-kserve-658676577-55vz9\" (UID: \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\") " pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" Nov 02 21:22:51.383192 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:51.383159 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-tmp-dir\") pod \"simulator-w2-6fa533-kserve-658676577-55vz9\" (UID: \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\") " pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" Nov 02 21:22:51.383421 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:51.383208 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2knmw\" (UniqueName: \"kubernetes.io/projected/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-kube-api-access-2knmw\") pod \"simulator-w2-6fa533-kserve-658676577-55vz9\" (UID: \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\") " pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" Nov 02 21:22:51.383421 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:51.383233 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-model-cache\") pod \"simulator-w2-6fa533-kserve-658676577-55vz9\" (UID: \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\") " pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" Nov 02 21:22:51.383421 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:51.383248 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-tls-certs\") pod \"simulator-w2-6fa533-kserve-658676577-55vz9\" (UID: \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\") " pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" Nov 02 21:22:51.383421 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:51.383335 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-dshm\") pod \"simulator-w2-6fa533-kserve-658676577-55vz9\" (UID: \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\") " pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" Nov 02 21:22:51.383604 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:51.383568 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-home\") pod \"simulator-w2-6fa533-kserve-658676577-55vz9\" (UID: \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\") " pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" Nov 02 21:22:51.383673 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:51.383654 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-model-cache\") pod \"simulator-w2-6fa533-kserve-658676577-55vz9\" (UID: \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\") " pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" Nov 02 21:22:51.383737 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:51.383701 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-tmp-dir\") pod \"simulator-w2-6fa533-kserve-658676577-55vz9\" (UID: \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\") " pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" Nov 02 21:22:51.385659 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:51.385636 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-dshm\") pod \"simulator-w2-6fa533-kserve-658676577-55vz9\" (UID: \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\") " pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" Nov 02 21:22:51.385987 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:51.385968 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-tls-certs\") pod \"simulator-w2-6fa533-kserve-658676577-55vz9\" (UID: \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\") " pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" Nov 02 21:22:51.437618 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:51.437587 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2knmw\" (UniqueName: \"kubernetes.io/projected/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-kube-api-access-2knmw\") pod \"simulator-w2-6fa533-kserve-658676577-55vz9\" (UID: \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\") " pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" Nov 02 21:22:51.446314 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:51.446281 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" Nov 02 21:22:51.821953 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:51.821930 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9"] Nov 02 21:22:51.823852 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:22:51.823821 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d6a5790_dd1f_4721_bbde_3aa6b14999b8.slice/crio-b0c217839ee5af527b55c676c5eee70dce2674788248b811fe44eecd41b43944 WatchSource:0}: Error finding container b0c217839ee5af527b55c676c5eee70dce2674788248b811fe44eecd41b43944: Status 404 returned error can't find the container with id b0c217839ee5af527b55c676c5eee70dce2674788248b811fe44eecd41b43944 Nov 02 21:22:52.438310 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:52.438263 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:52.521193 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:52.521160 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" event={"ID":"1d6a5790-dd1f-4721-bbde-3aa6b14999b8","Type":"ContainerStarted","Data":"6b90bbe4ba8d433cc5a63e6a4e7b3aa6b3d3a0634fbe47cd783d79e51a780003"} Nov 02 21:22:52.521366 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:52.521205 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" event={"ID":"1d6a5790-dd1f-4721-bbde-3aa6b14999b8","Type":"ContainerStarted","Data":"b0c217839ee5af527b55c676c5eee70dce2674788248b811fe44eecd41b43944"} Nov 02 21:22:52.629863 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:52.629806 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" podStartSLOduration=1.629792347 podStartE2EDuration="1.629792347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:22:52.62708322 +0000 UTC m=+865.066657937" watchObservedRunningTime="2026-11-02 21:22:52.629792347 +0000 UTC m=+865.069367070" Nov 02 21:22:53.447281 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:53.447246 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" Nov 02 21:22:57.429567 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:22:57.429526 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:22:58.473527 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:22:58.473488 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:23:01.446873 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:01.446829 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" Nov 02 21:23:01.459841 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:01.459811 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" Nov 02 21:23:01.584392 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:01.584361 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" Nov 02 21:23:02.443583 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:02.443528 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:23:05.881087 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:05.881048 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:23:07.523188 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:07.523121 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:23:12.630428 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:12.630393 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:23:17.238324 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:17.238290 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:23:17.745016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:17.744980 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:23:22.658644 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:22.658611 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:23:22.863953 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:22.863915 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:23:27.240914 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.240874 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf"] Nov 02 21:23:27.244978 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.244953 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" Nov 02 21:23:27.295305 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.295273 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0c9014\"/\"premium-w0-0c9014-kserve-self-signed-certs\"" Nov 02 21:23:27.298640 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.298606 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf"] Nov 02 21:23:27.335592 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.335557 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td6xr\" (UniqueName: \"kubernetes.io/projected/7cb3ac21-f737-4fc5-927e-4a8388582812-kube-api-access-td6xr\") pod \"premium-w0-0c9014-kserve-75d9c897cc-7ttjf\" (UID: \"7cb3ac21-f737-4fc5-927e-4a8388582812\") " pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" Nov 02 21:23:27.335780 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.335610 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cb3ac21-f737-4fc5-927e-4a8388582812-dshm\") pod \"premium-w0-0c9014-kserve-75d9c897cc-7ttjf\" (UID: \"7cb3ac21-f737-4fc5-927e-4a8388582812\") " pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" Nov 02 21:23:27.335780 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.335637 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cb3ac21-f737-4fc5-927e-4a8388582812-tls-certs\") pod \"premium-w0-0c9014-kserve-75d9c897cc-7ttjf\" (UID: \"7cb3ac21-f737-4fc5-927e-4a8388582812\") " pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" Nov 02 21:23:27.335780 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.335742 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cb3ac21-f737-4fc5-927e-4a8388582812-home\") pod \"premium-w0-0c9014-kserve-75d9c897cc-7ttjf\" (UID: \"7cb3ac21-f737-4fc5-927e-4a8388582812\") " pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" Nov 02 21:23:27.335913 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.335788 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cb3ac21-f737-4fc5-927e-4a8388582812-model-cache\") pod \"premium-w0-0c9014-kserve-75d9c897cc-7ttjf\" (UID: \"7cb3ac21-f737-4fc5-927e-4a8388582812\") " pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" Nov 02 21:23:27.335913 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.335881 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cb3ac21-f737-4fc5-927e-4a8388582812-tmp-dir\") pod \"premium-w0-0c9014-kserve-75d9c897cc-7ttjf\" (UID: \"7cb3ac21-f737-4fc5-927e-4a8388582812\") " pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" Nov 02 21:23:27.405159 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.405098 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:23:27.436676 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.436630 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td6xr\" (UniqueName: \"kubernetes.io/projected/7cb3ac21-f737-4fc5-927e-4a8388582812-kube-api-access-td6xr\") pod \"premium-w0-0c9014-kserve-75d9c897cc-7ttjf\" (UID: \"7cb3ac21-f737-4fc5-927e-4a8388582812\") " pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" Nov 02 21:23:27.436871 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.436693 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cb3ac21-f737-4fc5-927e-4a8388582812-dshm\") pod \"premium-w0-0c9014-kserve-75d9c897cc-7ttjf\" (UID: \"7cb3ac21-f737-4fc5-927e-4a8388582812\") " pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" Nov 02 21:23:27.436871 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.436714 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cb3ac21-f737-4fc5-927e-4a8388582812-tls-certs\") pod \"premium-w0-0c9014-kserve-75d9c897cc-7ttjf\" (UID: \"7cb3ac21-f737-4fc5-927e-4a8388582812\") " pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" Nov 02 21:23:27.436871 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.436736 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cb3ac21-f737-4fc5-927e-4a8388582812-home\") pod \"premium-w0-0c9014-kserve-75d9c897cc-7ttjf\" (UID: \"7cb3ac21-f737-4fc5-927e-4a8388582812\") " pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" Nov 02 21:23:27.436871 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.436758 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cb3ac21-f737-4fc5-927e-4a8388582812-model-cache\") pod \"premium-w0-0c9014-kserve-75d9c897cc-7ttjf\" (UID: \"7cb3ac21-f737-4fc5-927e-4a8388582812\") " pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" Nov 02 21:23:27.436871 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.436804 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cb3ac21-f737-4fc5-927e-4a8388582812-tmp-dir\") pod \"premium-w0-0c9014-kserve-75d9c897cc-7ttjf\" (UID: \"7cb3ac21-f737-4fc5-927e-4a8388582812\") " pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" Nov 02 21:23:27.437292 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.437244 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cb3ac21-f737-4fc5-927e-4a8388582812-home\") pod \"premium-w0-0c9014-kserve-75d9c897cc-7ttjf\" (UID: \"7cb3ac21-f737-4fc5-927e-4a8388582812\") " pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" Nov 02 21:23:27.437439 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.437267 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cb3ac21-f737-4fc5-927e-4a8388582812-tmp-dir\") pod \"premium-w0-0c9014-kserve-75d9c897cc-7ttjf\" (UID: \"7cb3ac21-f737-4fc5-927e-4a8388582812\") " pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" Nov 02 21:23:27.437439 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.437345 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cb3ac21-f737-4fc5-927e-4a8388582812-model-cache\") pod \"premium-w0-0c9014-kserve-75d9c897cc-7ttjf\" (UID: \"7cb3ac21-f737-4fc5-927e-4a8388582812\") " pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" Nov 02 21:23:27.439140 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.439102 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cb3ac21-f737-4fc5-927e-4a8388582812-dshm\") pod \"premium-w0-0c9014-kserve-75d9c897cc-7ttjf\" (UID: \"7cb3ac21-f737-4fc5-927e-4a8388582812\") " pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" Nov 02 21:23:27.439396 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.439379 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cb3ac21-f737-4fc5-927e-4a8388582812-tls-certs\") pod \"premium-w0-0c9014-kserve-75d9c897cc-7ttjf\" (UID: \"7cb3ac21-f737-4fc5-927e-4a8388582812\") " pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" Nov 02 21:23:27.490329 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.490294 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td6xr\" (UniqueName: \"kubernetes.io/projected/7cb3ac21-f737-4fc5-927e-4a8388582812-kube-api-access-td6xr\") pod \"premium-w0-0c9014-kserve-75d9c897cc-7ttjf\" (UID: \"7cb3ac21-f737-4fc5-927e-4a8388582812\") " pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" Nov 02 21:23:27.557736 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.557699 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" Nov 02 21:23:27.744730 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.744692 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:23:27.924005 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.923931 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 21:23:27.925182 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.925161 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 21:23:27.932552 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.932531 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 21:23:27.933708 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.933691 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 21:23:27.940912 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:27.940895 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf"] Nov 02 21:23:27.943422 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:23:27.943398 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cb3ac21_f737_4fc5_927e_4a8388582812.slice/crio-5af46704022743e30370b8a04a98a56293d8d6af2826135e6951dbc918b354a7 WatchSource:0}: Error finding container 5af46704022743e30370b8a04a98a56293d8d6af2826135e6951dbc918b354a7: Status 404 returned error can't find the container with id 5af46704022743e30370b8a04a98a56293d8d6af2826135e6951dbc918b354a7 Nov 02 21:23:28.472800 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:23:28.472759 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:23:28.732412 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:28.732324 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" event={"ID":"7cb3ac21-f737-4fc5-927e-4a8388582812","Type":"ContainerStarted","Data":"9fdb74d9d26777d29405d11d86f7f00945bcf08258515a370baae8069c37f14b"} Nov 02 21:23:28.732412 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:28.732372 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" event={"ID":"7cb3ac21-f737-4fc5-927e-4a8388582812","Type":"ContainerStarted","Data":"5af46704022743e30370b8a04a98a56293d8d6af2826135e6951dbc918b354a7"} Nov 02 21:23:28.841091 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:28.841031 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" podStartSLOduration=1.841015227 podStartE2EDuration="1.841015227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:23:28.8396578 +0000 UTC m=+901.279232522" watchObservedRunningTime="2026-11-02 21:23:28.841015227 +0000 UTC m=+901.280589949" Nov 02 21:23:29.558294 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:29.558254 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" Nov 02 21:23:32.857053 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:32.857013 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:23:37.558482 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:37.558439 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" Nov 02 21:23:37.571216 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:37.571190 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" Nov 02 21:23:37.792504 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:37.792474 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" Nov 02 21:23:37.833609 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:37.833465 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:23:42.936409 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:42.936353 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:23:47.941838 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:47.941797 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:23:53.131232 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:53.131189 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:23:54.144272 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:54.144231 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:23:58.124190 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:23:58.124126 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:23:58.468714 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:23:58.468619 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:24:03.130642 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:24:03.130602 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:08.172801 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:24:08.172725 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:13.210159 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:24:13.210111 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:14.486928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:24:14.486878 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:18.340514 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:24:18.340475 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:22.669754 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:24:22.669718 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:23.237050 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:24:23.237010 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:28.320883 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:24:28.320837 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:28.470408 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:24:28.470361 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:24:33.464156 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:24:33.464104 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:38.539406 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:24:38.539363 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:43.638647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:24:43.638607 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:48.635533 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:24:48.635495 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:53.642680 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:24:53.642641 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:24:58.468879 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:24:58.468835 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:24:58.625558 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:24:58.625524 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:03.706627 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:03.706586 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:06.313824 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:06.313780 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:06.516245 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:06.516208 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:06.737427 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:06.737337 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:08.800751 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:08.800718 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:10.266846 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:10.266807 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:13.921441 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:13.921403 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:19.031470 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:19.031430 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:22.997549 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:22.997513 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:24.135710 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:24.135652 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:24.946749 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:24.946707 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z"] Nov 02 21:25:24.951295 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:24.951260 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" Nov 02 21:25:25.002959 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:25.002926 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z"] Nov 02 21:25:25.030066 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:25.030013 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fbd3bf6-3386-414e-8a35-240ba641d306-tmp-dir\") pod \"e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z\" (UID: \"8fbd3bf6-3386-414e-8a35-240ba641d306\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" Nov 02 21:25:25.030276 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:25.030075 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fbd3bf6-3386-414e-8a35-240ba641d306-model-cache\") pod \"e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z\" (UID: \"8fbd3bf6-3386-414e-8a35-240ba641d306\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" Nov 02 21:25:25.030276 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:25.030194 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fbd3bf6-3386-414e-8a35-240ba641d306-tls-certs\") pod \"e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z\" (UID: \"8fbd3bf6-3386-414e-8a35-240ba641d306\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" Nov 02 21:25:25.030276 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:25.030254 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lngp4\" (UniqueName: \"kubernetes.io/projected/8fbd3bf6-3386-414e-8a35-240ba641d306-kube-api-access-lngp4\") pod \"e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z\" (UID: \"8fbd3bf6-3386-414e-8a35-240ba641d306\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" Nov 02 21:25:25.030467 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:25.030351 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fbd3bf6-3386-414e-8a35-240ba641d306-home\") pod \"e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z\" (UID: \"8fbd3bf6-3386-414e-8a35-240ba641d306\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" Nov 02 21:25:25.030467 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:25.030402 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fbd3bf6-3386-414e-8a35-240ba641d306-dshm\") pod \"e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z\" (UID: \"8fbd3bf6-3386-414e-8a35-240ba641d306\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" Nov 02 21:25:25.045611 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:25.045580 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d6cb38\"/\"e2e-conflict-a-00e30839-kserve-self-signed-certs\"" Nov 02 21:25:25.130563 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:25.130508 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:25.131229 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:25.131204 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fbd3bf6-3386-414e-8a35-240ba641d306-model-cache\") pod \"e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z\" (UID: \"8fbd3bf6-3386-414e-8a35-240ba641d306\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" Nov 02 21:25:25.131316 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:25.131272 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fbd3bf6-3386-414e-8a35-240ba641d306-tls-certs\") pod \"e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z\" (UID: \"8fbd3bf6-3386-414e-8a35-240ba641d306\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" Nov 02 21:25:25.131381 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:25.131341 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lngp4\" (UniqueName: \"kubernetes.io/projected/8fbd3bf6-3386-414e-8a35-240ba641d306-kube-api-access-lngp4\") pod \"e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z\" (UID: \"8fbd3bf6-3386-414e-8a35-240ba641d306\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" Nov 02 21:25:25.131435 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:25.131390 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fbd3bf6-3386-414e-8a35-240ba641d306-home\") pod \"e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z\" (UID: \"8fbd3bf6-3386-414e-8a35-240ba641d306\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" Nov 02 21:25:25.131435 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:25.131426 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fbd3bf6-3386-414e-8a35-240ba641d306-dshm\") pod \"e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z\" (UID: \"8fbd3bf6-3386-414e-8a35-240ba641d306\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" Nov 02 21:25:25.131542 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:25.131527 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fbd3bf6-3386-414e-8a35-240ba641d306-tmp-dir\") pod \"e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z\" (UID: \"8fbd3bf6-3386-414e-8a35-240ba641d306\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" Nov 02 21:25:25.131706 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:25.131676 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fbd3bf6-3386-414e-8a35-240ba641d306-model-cache\") pod \"e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z\" (UID: \"8fbd3bf6-3386-414e-8a35-240ba641d306\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" Nov 02 21:25:25.131903 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:25.131878 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fbd3bf6-3386-414e-8a35-240ba641d306-tmp-dir\") pod \"e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z\" (UID: \"8fbd3bf6-3386-414e-8a35-240ba641d306\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" Nov 02 21:25:25.132094 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:25.132067 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fbd3bf6-3386-414e-8a35-240ba641d306-home\") pod \"e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z\" (UID: \"8fbd3bf6-3386-414e-8a35-240ba641d306\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" Nov 02 21:25:25.134288 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:25.134264 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fbd3bf6-3386-414e-8a35-240ba641d306-dshm\") pod \"e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z\" (UID: \"8fbd3bf6-3386-414e-8a35-240ba641d306\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" Nov 02 21:25:25.134683 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:25.134662 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fbd3bf6-3386-414e-8a35-240ba641d306-tls-certs\") pod \"e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z\" (UID: \"8fbd3bf6-3386-414e-8a35-240ba641d306\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" Nov 02 21:25:25.185954 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:25.185907 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lngp4\" (UniqueName: \"kubernetes.io/projected/8fbd3bf6-3386-414e-8a35-240ba641d306-kube-api-access-lngp4\") pod \"e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z\" (UID: \"8fbd3bf6-3386-414e-8a35-240ba641d306\") " pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" Nov 02 21:25:25.265628 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:25.265580 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" Nov 02 21:25:25.692010 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:25.691981 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z"] Nov 02 21:25:25.695460 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:25:25.695422 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fbd3bf6_3386_414e_8a35_240ba641d306.slice/crio-4acb6fcee5a7b02075719bc3933eeeb4724318000b849c312a28aa74828105fe WatchSource:0}: Error finding container 4acb6fcee5a7b02075719bc3933eeeb4724318000b849c312a28aa74828105fe: Status 404 returned error can't find the container with id 4acb6fcee5a7b02075719bc3933eeeb4724318000b849c312a28aa74828105fe Nov 02 21:25:26.449933 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:26.449897 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" event={"ID":"8fbd3bf6-3386-414e-8a35-240ba641d306","Type":"ContainerStarted","Data":"b5881924a6ebd8417184685813e6b0b935fd0a874f39243a58c9f991fb6b2690"} Nov 02 21:25:26.450485 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:26.449946 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" event={"ID":"8fbd3bf6-3386-414e-8a35-240ba641d306","Type":"ContainerStarted","Data":"4acb6fcee5a7b02075719bc3933eeeb4724318000b849c312a28aa74828105fe"} Nov 02 21:25:27.268928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:27.268884 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" Nov 02 21:25:28.469944 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:25:28.469899 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:25:30.455273 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:30.455230 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:34.425237 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:34.425175 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:35.265868 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:35.265837 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" Nov 02 21:25:35.278893 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:35.278865 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" Nov 02 21:25:35.386451 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:35.386401 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" podStartSLOduration=11.386387836 podStartE2EDuration="11.386387836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:25:26.57341054 +0000 UTC m=+1019.012985262" watchObservedRunningTime="2026-11-02 21:25:35.386387836 +0000 UTC m=+1027.825962557" Nov 02 21:25:35.536751 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:35.536670 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" Nov 02 21:25:39.517121 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:39.517008 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:44.523495 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:44.523453 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:46.066993 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.066961 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9"] Nov 02 21:25:46.070838 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.070803 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.199859 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.199825 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gljp9\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.200011 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.199875 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gljp9\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.200064 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.200016 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gljp9\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.200101 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.200088 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gljp9\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.200158 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.200115 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtcxj\" (UniqueName: \"kubernetes.io/projected/8c71c82d-68ca-434b-ae75-6b9630d5cb46-kube-api-access-mtcxj\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gljp9\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.200210 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.200166 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gljp9\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.200284 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.200257 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gljp9\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.200341 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.200324 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gljp9\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.200407 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.200389 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gljp9\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.256051 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.256020 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9"] Nov 02 21:25:46.301334 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.301299 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gljp9\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.301500 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.301343 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gljp9\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.301500 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.301392 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gljp9\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.301500 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.301414 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gljp9\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.301500 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.301451 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gljp9\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.301712 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.301641 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gljp9\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.301712 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.301661 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtcxj\" (UniqueName: \"kubernetes.io/projected/8c71c82d-68ca-434b-ae75-6b9630d5cb46-kube-api-access-mtcxj\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gljp9\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.301712 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.301682 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gljp9\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.301712 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.301709 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gljp9\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.301924 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.301776 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gljp9\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.302065 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.302039 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gljp9\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.302172 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.302070 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gljp9\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.302172 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.302090 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gljp9\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.302356 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.302333 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gljp9\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.303731 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.303709 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gljp9\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.304007 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.303990 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gljp9\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.354712 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.354617 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gljp9\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.356045 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.356018 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtcxj\" (UniqueName: \"kubernetes.io/projected/8c71c82d-68ca-434b-ae75-6b9630d5cb46-kube-api-access-mtcxj\") pod \"maas-default-gateway-openshift-default-fcb75c45f-gljp9\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.384399 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.384369 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:46.805731 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.805706 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9"] Nov 02 21:25:46.808446 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:25:46.808410 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c71c82d_68ca_434b_ae75_6b9630d5cb46.slice/crio-17289505c687190604b2159c328a71c5945f734b284425d6d287df98c88a4618 WatchSource:0}: Error finding container 17289505c687190604b2159c328a71c5945f734b284425d6d287df98c88a4618: Status 404 returned error can't find the container with id 17289505c687190604b2159c328a71c5945f734b284425d6d287df98c88a4618 Nov 02 21:25:46.810873 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.810841 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:25:46.810958 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.810907 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:25:46.810958 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:46.810937 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:25:47.650162 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:47.649274 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" event={"ID":"8c71c82d-68ca-434b-ae75-6b9630d5cb46","Type":"ContainerStarted","Data":"668a88949c3d6186014c9760e7c7e6a8bbc30c5bae4b92571027f5e841d84dd6"} Nov 02 21:25:47.650162 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:47.649343 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" event={"ID":"8c71c82d-68ca-434b-ae75-6b9630d5cb46","Type":"ContainerStarted","Data":"17289505c687190604b2159c328a71c5945f734b284425d6d287df98c88a4618"} Nov 02 21:25:47.816268 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:47.816217 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" podStartSLOduration=1.816202777 podStartE2EDuration="1.816202777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:25:47.81347489 +0000 UTC m=+1040.253049613" watchObservedRunningTime="2026-11-02 21:25:47.816202777 +0000 UTC m=+1040.255777499" Nov 02 21:25:48.384940 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:48.384902 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:48.386168 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:48.386123 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gljp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 02 21:25:48.386224 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:48.386195 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" podUID="8c71c82d-68ca-434b-ae75-6b9630d5cb46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 02 21:25:49.385358 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:49.385317 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gljp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 02 21:25:49.385872 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:49.385384 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" podUID="8c71c82d-68ca-434b-ae75-6b9630d5cb46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 02 21:25:49.706670 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:49.706581 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:50.384705 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:50.384671 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gljp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 02 21:25:50.384893 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:50.384729 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" podUID="8c71c82d-68ca-434b-ae75-6b9630d5cb46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 02 21:25:50.861657 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:50.861619 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:51.066416 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:51.066379 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:51.385352 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:51.385316 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gljp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 02 21:25:51.385528 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:51.385371 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" podUID="8c71c82d-68ca-434b-ae75-6b9630d5cb46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 02 21:25:52.385158 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:52.385097 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gljp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 02 21:25:52.385651 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:52.385177 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" podUID="8c71c82d-68ca-434b-ae75-6b9630d5cb46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 02 21:25:53.385034 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:53.384998 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gljp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 02 21:25:53.385237 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:53.385051 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" podUID="8c71c82d-68ca-434b-ae75-6b9630d5cb46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 02 21:25:54.384889 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:54.384853 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-gljp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 02 21:25:54.385087 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:54.384908 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" podUID="8c71c82d-68ca-434b-ae75-6b9630d5cb46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 02 21:25:54.785760 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:54.785716 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:54.999587 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:54.999544 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:25:55.430593 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:55.430554 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:55.430929 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:55.430897 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:55.431933 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:55.431914 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:25:58.474302 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:25:58.474265 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:25:59.843885 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:25:59.843850 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:01.103461 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:01.103422 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx"] Nov 02 21:26:01.108596 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:01.108571 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" Nov 02 21:26:01.184269 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:01.184238 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2wvs\" (UniqueName: \"kubernetes.io/projected/39b34814-d9a5-4505-9906-cd18e55eed4b-kube-api-access-b2wvs\") pod \"distinct-w1-5f765a-kserve-6f566d44d-wcvrx\" (UID: \"39b34814-d9a5-4505-9906-cd18e55eed4b\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" Nov 02 21:26:01.184450 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:01.184292 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39b34814-d9a5-4505-9906-cd18e55eed4b-model-cache\") pod \"distinct-w1-5f765a-kserve-6f566d44d-wcvrx\" (UID: \"39b34814-d9a5-4505-9906-cd18e55eed4b\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" Nov 02 21:26:01.184450 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:01.184435 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39b34814-d9a5-4505-9906-cd18e55eed4b-home\") pod \"distinct-w1-5f765a-kserve-6f566d44d-wcvrx\" (UID: \"39b34814-d9a5-4505-9906-cd18e55eed4b\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" Nov 02 21:26:01.184576 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:01.184476 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b34814-d9a5-4505-9906-cd18e55eed4b-tmp-dir\") pod \"distinct-w1-5f765a-kserve-6f566d44d-wcvrx\" (UID: \"39b34814-d9a5-4505-9906-cd18e55eed4b\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" Nov 02 21:26:01.184576 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:01.184508 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b34814-d9a5-4505-9906-cd18e55eed4b-tls-certs\") pod \"distinct-w1-5f765a-kserve-6f566d44d-wcvrx\" (UID: \"39b34814-d9a5-4505-9906-cd18e55eed4b\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" Nov 02 21:26:01.184640 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:01.184592 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39b34814-d9a5-4505-9906-cd18e55eed4b-dshm\") pod \"distinct-w1-5f765a-kserve-6f566d44d-wcvrx\" (UID: \"39b34814-d9a5-4505-9906-cd18e55eed4b\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" Nov 02 21:26:01.203473 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:01.203448 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5f765a\"/\"kube-root-ca.crt\"" Nov 02 21:26:01.203682 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:01.203663 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5f765a\"/\"openshift-service-ca.crt\"" Nov 02 21:26:01.203682 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:01.203677 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5f765a\"/\"distinct-w1-5f765a-kserve-self-signed-certs\"" Nov 02 21:26:01.203804 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:01.203689 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5f765a\"/\"default-dockercfg-tpxs7\"" Nov 02 21:26:01.206419 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:01.206398 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx"] Nov 02 21:26:01.268613 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:01.268580 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:01.285158 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:01.285122 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39b34814-d9a5-4505-9906-cd18e55eed4b-model-cache\") pod \"distinct-w1-5f765a-kserve-6f566d44d-wcvrx\" (UID: \"39b34814-d9a5-4505-9906-cd18e55eed4b\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" Nov 02 21:26:01.285284 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:01.285209 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39b34814-d9a5-4505-9906-cd18e55eed4b-home\") pod \"distinct-w1-5f765a-kserve-6f566d44d-wcvrx\" (UID: \"39b34814-d9a5-4505-9906-cd18e55eed4b\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" Nov 02 21:26:01.285284 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:01.285228 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b34814-d9a5-4505-9906-cd18e55eed4b-tmp-dir\") pod \"distinct-w1-5f765a-kserve-6f566d44d-wcvrx\" (UID: \"39b34814-d9a5-4505-9906-cd18e55eed4b\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" Nov 02 21:26:01.285284 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:01.285243 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b34814-d9a5-4505-9906-cd18e55eed4b-tls-certs\") pod \"distinct-w1-5f765a-kserve-6f566d44d-wcvrx\" (UID: \"39b34814-d9a5-4505-9906-cd18e55eed4b\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" Nov 02 21:26:01.285284 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:01.285259 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39b34814-d9a5-4505-9906-cd18e55eed4b-dshm\") pod \"distinct-w1-5f765a-kserve-6f566d44d-wcvrx\" (UID: \"39b34814-d9a5-4505-9906-cd18e55eed4b\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" Nov 02 21:26:01.285507 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:01.285351 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2wvs\" (UniqueName: \"kubernetes.io/projected/39b34814-d9a5-4505-9906-cd18e55eed4b-kube-api-access-b2wvs\") pod \"distinct-w1-5f765a-kserve-6f566d44d-wcvrx\" (UID: \"39b34814-d9a5-4505-9906-cd18e55eed4b\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" Nov 02 21:26:01.285565 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:01.285518 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39b34814-d9a5-4505-9906-cd18e55eed4b-model-cache\") pod \"distinct-w1-5f765a-kserve-6f566d44d-wcvrx\" (UID: \"39b34814-d9a5-4505-9906-cd18e55eed4b\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" Nov 02 21:26:01.285625 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:01.285582 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39b34814-d9a5-4505-9906-cd18e55eed4b-home\") pod \"distinct-w1-5f765a-kserve-6f566d44d-wcvrx\" (UID: \"39b34814-d9a5-4505-9906-cd18e55eed4b\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" Nov 02 21:26:01.285663 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:01.285628 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b34814-d9a5-4505-9906-cd18e55eed4b-tmp-dir\") pod \"distinct-w1-5f765a-kserve-6f566d44d-wcvrx\" (UID: \"39b34814-d9a5-4505-9906-cd18e55eed4b\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" Nov 02 21:26:01.287981 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:01.287952 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39b34814-d9a5-4505-9906-cd18e55eed4b-dshm\") pod \"distinct-w1-5f765a-kserve-6f566d44d-wcvrx\" (UID: \"39b34814-d9a5-4505-9906-cd18e55eed4b\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" Nov 02 21:26:01.288303 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:01.288275 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b34814-d9a5-4505-9906-cd18e55eed4b-tls-certs\") pod \"distinct-w1-5f765a-kserve-6f566d44d-wcvrx\" (UID: \"39b34814-d9a5-4505-9906-cd18e55eed4b\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" Nov 02 21:26:01.339281 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:01.339249 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2wvs\" (UniqueName: \"kubernetes.io/projected/39b34814-d9a5-4505-9906-cd18e55eed4b-kube-api-access-b2wvs\") pod \"distinct-w1-5f765a-kserve-6f566d44d-wcvrx\" (UID: \"39b34814-d9a5-4505-9906-cd18e55eed4b\") " pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" Nov 02 21:26:01.421755 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:01.421665 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" Nov 02 21:26:01.826489 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:01.826465 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx"] Nov 02 21:26:01.828736 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:26:01.828704 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39b34814_d9a5_4505_9906_cd18e55eed4b.slice/crio-d07688cd524991ccf332308b8d5c99e2b0ef97faec5363eea9bc0fd01c6ab289 WatchSource:0}: Error finding container d07688cd524991ccf332308b8d5c99e2b0ef97faec5363eea9bc0fd01c6ab289: Status 404 returned error can't find the container with id d07688cd524991ccf332308b8d5c99e2b0ef97faec5363eea9bc0fd01c6ab289 Nov 02 21:26:02.747324 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:02.747288 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" event={"ID":"39b34814-d9a5-4505-9906-cd18e55eed4b","Type":"ContainerStarted","Data":"a3f3ac2bbdd411506337e19d11884b0c3780c7652f4be7ddcb28c55e89518075"} Nov 02 21:26:02.747324 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:02.747328 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" event={"ID":"39b34814-d9a5-4505-9906-cd18e55eed4b","Type":"ContainerStarted","Data":"d07688cd524991ccf332308b8d5c99e2b0ef97faec5363eea9bc0fd01c6ab289"} Nov 02 21:26:02.855777 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:02.855724 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" podStartSLOduration=1.8557101409999999 podStartE2EDuration="1.855710141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:26:02.853572376 +0000 UTC m=+1055.293147097" watchObservedRunningTime="2026-11-02 21:26:02.855710141 +0000 UTC m=+1055.295284888" Nov 02 21:26:03.422388 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:03.422354 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" Nov 02 21:26:04.854235 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:04.854200 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:05.935462 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:05.935425 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h"] Nov 02 21:26:05.939504 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:05.939478 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:05.988245 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:05.988216 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d6cb38-openshift-default-dockercfg-mbhbl\"" Nov 02 21:26:06.036726 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.036686 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:06.037837 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.036761 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:06.037837 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.036810 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:06.037837 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.036875 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:06.037837 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.036932 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:06.037837 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.036975 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:06.037837 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.037441 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:06.037837 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.037492 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srdww\" (UniqueName: \"kubernetes.io/projected/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-kube-api-access-srdww\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:06.037837 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.037544 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:06.038239 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.038085 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h"] Nov 02 21:26:06.138887 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.138853 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:06.139080 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.138911 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:06.139080 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.138971 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:06.139232 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.139195 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:06.139315 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.139270 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:06.139374 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.139317 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:06.139374 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.139355 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srdww\" (UniqueName: \"kubernetes.io/projected/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-kube-api-access-srdww\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:06.139482 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.139407 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:06.139482 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.139436 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:06.139482 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.139463 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:06.140252 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.139752 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:06.140252 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.140102 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:06.140252 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.140199 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:06.140499 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.140444 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:06.142053 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.142021 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:06.142263 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.142244 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:06.192912 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.192839 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:06.193329 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.193307 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srdww\" (UniqueName: \"kubernetes.io/projected/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-kube-api-access-srdww\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:06.256057 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.256027 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:06.683737 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.683686 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h"] Nov 02 21:26:06.692832 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.692785 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:26:06.692951 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.692881 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:26:06.692951 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.692932 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:26:06.785153 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.781776 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" event={"ID":"52bdab02-9d8b-4a0f-90b2-bc9e705845f0","Type":"ContainerStarted","Data":"7c63c8074e004853198d0194932da0464e93aa3d39ef64e12b0e415231b87d88"} Nov 02 21:26:06.883309 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.883268 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79"] Nov 02 21:26:06.887327 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.887300 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:06.954208 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.951327 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-6fa533-openshift-default-dockercfg-psqwg\"" Nov 02 21:26:06.954208 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.952001 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:06.954208 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.952044 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:06.954208 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.952148 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:06.954208 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.952181 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5df21a5e-97fc-4564-8118-884d34b0d3d1-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:06.954208 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.952283 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:06.954208 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.952411 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5df21a5e-97fc-4564-8118-884d34b0d3d1-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:06.954208 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.952469 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5df21a5e-97fc-4564-8118-884d34b0d3d1-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:06.954208 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.952524 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8m6d\" (UniqueName: \"kubernetes.io/projected/5df21a5e-97fc-4564-8118-884d34b0d3d1-kube-api-access-j8m6d\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:06.954208 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.952570 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:06.954208 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:06.953252 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79"] Nov 02 21:26:07.060211 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:07.058687 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:07.060211 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:07.058783 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:07.060211 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:07.058813 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5df21a5e-97fc-4564-8118-884d34b0d3d1-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:07.060211 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:07.058839 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:07.060211 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:07.058885 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5df21a5e-97fc-4564-8118-884d34b0d3d1-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:07.060211 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:07.058930 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5df21a5e-97fc-4564-8118-884d34b0d3d1-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:07.060211 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:07.058976 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8m6d\" (UniqueName: \"kubernetes.io/projected/5df21a5e-97fc-4564-8118-884d34b0d3d1-kube-api-access-j8m6d\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:07.060211 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:07.059015 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:07.060211 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:07.059077 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:07.060211 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:07.059583 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:07.061387 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:07.061338 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5df21a5e-97fc-4564-8118-884d34b0d3d1-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:07.061711 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:07.061666 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:07.062107 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:07.062085 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:07.062389 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:07.062368 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:07.073681 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:07.073652 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:07.073880 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:07.073856 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5df21a5e-97fc-4564-8118-884d34b0d3d1-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:07.115536 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:07.115496 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5df21a5e-97fc-4564-8118-884d34b0d3d1-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:07.116119 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:07.116079 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8m6d\" (UniqueName: \"kubernetes.io/projected/5df21a5e-97fc-4564-8118-884d34b0d3d1-kube-api-access-j8m6d\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:07.203317 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:07.203282 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:07.704880 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:07.696513 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79"] Nov 02 21:26:07.708227 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:26:07.708186 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df21a5e_97fc_4564_8118_884d34b0d3d1.slice/crio-11ab4fef019663f83bc172198a3791de7d2aa2aa7a06a4aedbe12f0c36d4454e WatchSource:0}: Error finding container 11ab4fef019663f83bc172198a3791de7d2aa2aa7a06a4aedbe12f0c36d4454e: Status 404 returned error can't find the container with id 11ab4fef019663f83bc172198a3791de7d2aa2aa7a06a4aedbe12f0c36d4454e Nov 02 21:26:07.710783 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:07.710748 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:26:07.710883 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:07.710827 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:26:07.710883 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:07.710873 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:26:07.792963 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:07.792922 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" event={"ID":"5df21a5e-97fc-4564-8118-884d34b0d3d1","Type":"ContainerStarted","Data":"11ab4fef019663f83bc172198a3791de7d2aa2aa7a06a4aedbe12f0c36d4454e"} Nov 02 21:26:07.795588 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:07.795543 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" event={"ID":"52bdab02-9d8b-4a0f-90b2-bc9e705845f0","Type":"ContainerStarted","Data":"9ebbcd2dfa896bddf79e85b1719e7da5610fae83828dad7c0e2dbf5a39702362"} Nov 02 21:26:07.922660 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:07.922601 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" podStartSLOduration=2.922588008 podStartE2EDuration="2.922588008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:26:07.922328792 +0000 UTC m=+1060.361903515" watchObservedRunningTime="2026-11-02 21:26:07.922588008 +0000 UTC m=+1060.362162730" Nov 02 21:26:08.257547 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:08.257512 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:08.264178 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:08.259787 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 02 21:26:08.264178 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:08.259845 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" podUID="52bdab02-9d8b-4a0f-90b2-bc9e705845f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 02 21:26:08.810662 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:08.810621 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" event={"ID":"5df21a5e-97fc-4564-8118-884d34b0d3d1","Type":"ContainerStarted","Data":"eb99a3abce76100f7cf0fd5b6de44d27604ebd23aa951b3e32f952ce96124f7f"} Nov 02 21:26:08.942630 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:08.942562 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" podStartSLOduration=2.942544275 podStartE2EDuration="2.942544275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:26:08.939609193 +0000 UTC m=+1061.379183929" watchObservedRunningTime="2026-11-02 21:26:08.942544275 +0000 UTC m=+1061.382118979" Nov 02 21:26:09.204387 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:09.204309 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:09.205865 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:09.205838 4935 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 02 21:26:09.206018 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:09.205885 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" podUID="5df21a5e-97fc-4564-8118-884d34b0d3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 02 21:26:09.258976 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:09.258928 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 02 21:26:09.259603 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:09.259551 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" podUID="52bdab02-9d8b-4a0f-90b2-bc9e705845f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 02 21:26:10.202184 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:10.200330 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:10.204243 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:10.204197 4935 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 02 21:26:10.204405 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:10.204267 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" podUID="5df21a5e-97fc-4564-8118-884d34b0d3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 02 21:26:10.257231 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:10.257190 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 02 21:26:10.257550 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:10.257502 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" podUID="52bdab02-9d8b-4a0f-90b2-bc9e705845f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 02 21:26:11.204875 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:11.204388 4935 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 02 21:26:11.204875 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:11.204459 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" podUID="5df21a5e-97fc-4564-8118-884d34b0d3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 02 21:26:11.259194 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:11.259042 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 02 21:26:11.259194 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:11.259099 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" podUID="52bdab02-9d8b-4a0f-90b2-bc9e705845f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 02 21:26:11.433265 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:11.423995 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" Nov 02 21:26:11.710608 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:11.710568 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" Nov 02 21:26:12.204589 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:12.204549 4935 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 02 21:26:12.204923 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:12.204881 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" podUID="5df21a5e-97fc-4564-8118-884d34b0d3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 02 21:26:12.258906 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:12.258867 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 02 21:26:12.259218 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:12.259187 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" podUID="52bdab02-9d8b-4a0f-90b2-bc9e705845f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 02 21:26:12.458397 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:12.458288 4935 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" podUID="1d6a5790-dd1f-4721-bbde-3aa6b14999b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:26:12.750598 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:12.750497 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" Nov 02 21:26:13.232396 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:13.232360 4935 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 02 21:26:13.233016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:13.232956 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" podUID="5df21a5e-97fc-4564-8118-884d34b0d3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 02 21:26:13.256600 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:13.256487 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 02 21:26:13.256600 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:13.256543 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" podUID="52bdab02-9d8b-4a0f-90b2-bc9e705845f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 02 21:26:14.172813 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:14.172771 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:14.206044 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:14.205569 4935 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 02 21:26:14.206044 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:14.205633 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" podUID="5df21a5e-97fc-4564-8118-884d34b0d3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 02 21:26:14.263872 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:14.263745 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 02 21:26:14.263872 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:14.263809 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" podUID="52bdab02-9d8b-4a0f-90b2-bc9e705845f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 02 21:26:14.355166 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:14.354268 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" podUID="3920de75-9c3e-4826-8de5-22fcc6d50b0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:26:14.396291 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:14.396239 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" podUID="aab5db02-4890-45e1-bcf8-1c43c335a393" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:26:14.543156 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:14.542264 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" podUID="8fbd3bf6-3386-414e-8a35-240ba641d306" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:26:15.208067 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:15.208031 4935 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 02 21:26:15.208394 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:15.208350 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" podUID="5df21a5e-97fc-4564-8118-884d34b0d3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 02 21:26:15.786405 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:15.786361 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:15.792042 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:15.792016 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:15.796900 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:15.796878 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:26:16.256928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:16.256843 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:16.334161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:16.334102 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:16.334763 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:16.334737 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:16.335597 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:16.335578 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:26:16.760037 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:16.760000 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:16.960445 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:16.960409 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:17.157466 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:17.157432 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:18.335241 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:18.335204 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:18.541460 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:18.541420 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:21.332601 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:21.332569 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:23.338031 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:23.337996 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:26.390026 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:26.389982 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:28.511822 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:26:28.511772 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:26:31.658467 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:31.658417 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:36.665092 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:36.665059 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:36.908334 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:36.908280 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:41.718943 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:41.718903 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:47.016159 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:47.015835 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:52.227253 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.227169 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k"] Nov 02 21:26:52.230274 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.230237 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:52.231746 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.231724 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.306773 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.306742 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e1984c-0e41-4b67-b36d-7796b3267a76-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.306948 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.306777 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e1984c-0e41-4b67-b36d-7796b3267a76-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.306948 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.306814 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdjfx\" (UniqueName: \"kubernetes.io/projected/c5e1984c-0e41-4b67-b36d-7796b3267a76-kube-api-access-jdjfx\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.307063 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.306979 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.307063 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.307037 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.307198 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.307177 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.307257 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.307219 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.307257 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.307241 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e1984c-0e41-4b67-b36d-7796b3267a76-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.307323 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.307266 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.408421 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.408385 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdjfx\" (UniqueName: \"kubernetes.io/projected/c5e1984c-0e41-4b67-b36d-7796b3267a76-kube-api-access-jdjfx\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.408602 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.408454 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.408602 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.408499 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.408602 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.408554 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.408602 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.408590 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.408819 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.408618 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e1984c-0e41-4b67-b36d-7796b3267a76-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.408819 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.408661 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.408819 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.408739 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e1984c-0e41-4b67-b36d-7796b3267a76-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.408819 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.408770 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e1984c-0e41-4b67-b36d-7796b3267a76-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.409018 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.408900 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.409018 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.408929 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.409018 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.408954 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.409229 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.409015 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.409490 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.409467 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e1984c-0e41-4b67-b36d-7796b3267a76-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.411047 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.411027 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.411271 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.411250 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e1984c-0e41-4b67-b36d-7796b3267a76-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.418253 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.418221 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k"] Nov 02 21:26:52.462382 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.462344 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e1984c-0e41-4b67-b36d-7796b3267a76-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.462511 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.462415 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdjfx\" (UniqueName: \"kubernetes.io/projected/c5e1984c-0e41-4b67-b36d-7796b3267a76-kube-api-access-jdjfx\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.545628 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.545599 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:52.967004 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.966976 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k"] Nov 02 21:26:52.968045 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:26:52.968016 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e1984c_0e41_4b67_b36d_7796b3267a76.slice/crio-978e2be7527b1eefc5e8197d18de2650895f700b405a1744a226e2e250899460 WatchSource:0}: Error finding container 978e2be7527b1eefc5e8197d18de2650895f700b405a1744a226e2e250899460: Status 404 returned error can't find the container with id 978e2be7527b1eefc5e8197d18de2650895f700b405a1744a226e2e250899460 Nov 02 21:26:52.970522 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.970489 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:26:52.970605 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.970559 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:26:52.970605 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:52.970591 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:26:53.213239 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:53.213200 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" event={"ID":"c5e1984c-0e41-4b67-b36d-7796b3267a76","Type":"ContainerStarted","Data":"50dc4b37692cad6a49f984a128151fa1ed65b562774850b721836c8f29765ff1"} Nov 02 21:26:53.213415 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:53.213267 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" event={"ID":"c5e1984c-0e41-4b67-b36d-7796b3267a76","Type":"ContainerStarted","Data":"978e2be7527b1eefc5e8197d18de2650895f700b405a1744a226e2e250899460"} Nov 02 21:26:53.323034 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:53.321795 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" podStartSLOduration=1.321776933 podStartE2EDuration="1.321776933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:26:53.321372417 +0000 UTC m=+1105.760947143" watchObservedRunningTime="2026-11-02 21:26:53.321776933 +0000 UTC m=+1105.761351655" Nov 02 21:26:53.548545 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:53.548497 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:26:54.549635 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:54.549590 4935 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 21:26:54.550108 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:54.549654 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" podUID="c5e1984c-0e41-4b67-b36d-7796b3267a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 21:26:55.548158 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:55.547470 4935 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 21:26:55.548158 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:55.547549 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" podUID="c5e1984c-0e41-4b67-b36d-7796b3267a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 21:26:56.546577 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:56.546533 4935 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 21:26:56.547218 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:56.547182 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" podUID="c5e1984c-0e41-4b67-b36d-7796b3267a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 21:26:57.369653 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:57.369609 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:57.549496 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:57.549457 4935 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 21:26:57.550089 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:57.550058 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" podUID="c5e1984c-0e41-4b67-b36d-7796b3267a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 21:26:58.475505 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:26:58.475466 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:26:58.548654 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:58.548611 4935 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 21:26:58.548836 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:58.548689 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" podUID="c5e1984c-0e41-4b67-b36d-7796b3267a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 21:26:59.267123 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:59.267083 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:26:59.546571 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:59.546457 4935 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 21:26:59.546571 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:26:59.546523 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" podUID="c5e1984c-0e41-4b67-b36d-7796b3267a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 21:27:00.552109 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:00.552073 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:27:00.552615 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:00.552465 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:27:00.553404 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:00.553374 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 21:27:02.432422 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:02.432383 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:07.425983 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:07.425941 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:07.661600 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:07.661553 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:12.439996 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:12.439906 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:17.553880 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:17.553841 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:21.169661 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:21.169620 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:21.428353 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:21.428273 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:22.554647 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:22.554598 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:23.680655 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:23.680618 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:27.604934 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:27.604896 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:28.470506 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:27:28.470441 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:27:31.326784 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:31.326714 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:33.813749 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:33.813666 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:37.760963 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:37.760923 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:42.727631 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:42.727587 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:47.318687 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:47.318653 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:47.861384 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:47.861342 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:48.248650 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:48.248538 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:50.005417 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:50.005382 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j"] Nov 02 21:27:50.009342 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:50.009320 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" Nov 02 21:27:50.067359 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:50.067323 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7nxl\" (UniqueName: \"kubernetes.io/projected/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-kube-api-access-c7nxl\") pod \"distinct-2-w2-6fa533-kserve-7f6749b868-cc68j\" (UID: \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" Nov 02 21:27:50.067529 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:50.067419 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-model-cache\") pod \"distinct-2-w2-6fa533-kserve-7f6749b868-cc68j\" (UID: \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" Nov 02 21:27:50.067529 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:50.067474 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-tls-certs\") pod \"distinct-2-w2-6fa533-kserve-7f6749b868-cc68j\" (UID: \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" Nov 02 21:27:50.067661 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:50.067595 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-dshm\") pod \"distinct-2-w2-6fa533-kserve-7f6749b868-cc68j\" (UID: \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" Nov 02 21:27:50.067661 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:50.067627 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-home\") pod \"distinct-2-w2-6fa533-kserve-7f6749b868-cc68j\" (UID: \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" Nov 02 21:27:50.067768 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:50.067723 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-tmp-dir\") pod \"distinct-2-w2-6fa533-kserve-7f6749b868-cc68j\" (UID: \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" Nov 02 21:27:50.104947 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:50.104921 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6fa533\"/\"distinct-2-w2-6fa533-kserve-self-signed-certs\"" Nov 02 21:27:50.107769 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:50.107740 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j"] Nov 02 21:27:50.168117 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:50.168084 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:50.168586 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:50.168564 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-dshm\") pod \"distinct-2-w2-6fa533-kserve-7f6749b868-cc68j\" (UID: \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" Nov 02 21:27:50.168688 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:50.168594 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-home\") pod \"distinct-2-w2-6fa533-kserve-7f6749b868-cc68j\" (UID: \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" Nov 02 21:27:50.168688 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:50.168641 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-tmp-dir\") pod \"distinct-2-w2-6fa533-kserve-7f6749b868-cc68j\" (UID: \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" Nov 02 21:27:50.168688 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:50.168673 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7nxl\" (UniqueName: \"kubernetes.io/projected/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-kube-api-access-c7nxl\") pod \"distinct-2-w2-6fa533-kserve-7f6749b868-cc68j\" (UID: \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" Nov 02 21:27:50.168847 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:50.168739 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-model-cache\") pod \"distinct-2-w2-6fa533-kserve-7f6749b868-cc68j\" (UID: \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" Nov 02 21:27:50.168986 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:50.168811 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-tls-certs\") pod \"distinct-2-w2-6fa533-kserve-7f6749b868-cc68j\" (UID: \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" Nov 02 21:27:50.169195 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:50.169172 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-home\") pod \"distinct-2-w2-6fa533-kserve-7f6749b868-cc68j\" (UID: \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" Nov 02 21:27:50.169289 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:50.169222 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-model-cache\") pod \"distinct-2-w2-6fa533-kserve-7f6749b868-cc68j\" (UID: \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" Nov 02 21:27:50.169419 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:50.169400 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-tmp-dir\") pod \"distinct-2-w2-6fa533-kserve-7f6749b868-cc68j\" (UID: \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" Nov 02 21:27:50.171191 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:50.171164 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-dshm\") pod \"distinct-2-w2-6fa533-kserve-7f6749b868-cc68j\" (UID: \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" Nov 02 21:27:50.171584 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:50.171559 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-tls-certs\") pod \"distinct-2-w2-6fa533-kserve-7f6749b868-cc68j\" (UID: \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" Nov 02 21:27:50.222940 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:50.222904 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7nxl\" (UniqueName: \"kubernetes.io/projected/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-kube-api-access-c7nxl\") pod \"distinct-2-w2-6fa533-kserve-7f6749b868-cc68j\" (UID: \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\") " pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" Nov 02 21:27:50.321041 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:50.320983 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" Nov 02 21:27:50.767031 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:50.766993 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j"] Nov 02 21:27:50.767467 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:27:50.767441 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ba3a2e4_a47f_425a_8c65_5813ef0939c9.slice/crio-4940b2e68f2a913f82557e55133542b7baea6d621df78494a1fe0e5d09289e00 WatchSource:0}: Error finding container 4940b2e68f2a913f82557e55133542b7baea6d621df78494a1fe0e5d09289e00: Status 404 returned error can't find the container with id 4940b2e68f2a913f82557e55133542b7baea6d621df78494a1fe0e5d09289e00 Nov 02 21:27:51.607360 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:51.607319 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" event={"ID":"7ba3a2e4-a47f-425a-8c65-5813ef0939c9","Type":"ContainerStarted","Data":"e8ca277623431acab078d34790f59e15e624602b18d9857858383e4d3578ead6"} Nov 02 21:27:51.607743 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:51.607371 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" event={"ID":"7ba3a2e4-a47f-425a-8c65-5813ef0939c9","Type":"ContainerStarted","Data":"4940b2e68f2a913f82557e55133542b7baea6d621df78494a1fe0e5d09289e00"} Nov 02 21:27:51.717212 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:51.717159 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" podStartSLOduration=2.717145531 podStartE2EDuration="2.717145531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:27:51.714286103 +0000 UTC m=+1164.153860824" watchObservedRunningTime="2026-11-02 21:27:51.717145531 +0000 UTC m=+1164.156720247" Nov 02 21:27:52.321078 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:52.321047 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" Nov 02 21:27:52.841892 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:52.841845 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:53.933840 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:53.933802 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z"] Nov 02 21:27:54.005709 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:54.005674 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:57.909982 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:27:57.909938 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:27:58.470199 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:27:58.470151 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:28:00.321625 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:00.321585 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" Nov 02 21:28:00.342582 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:00.342541 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" Nov 02 21:28:00.702722 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:00.702629 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" Nov 02 21:28:02.940179 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:02.940104 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:03.631207 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:03.631170 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:08.027956 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:08.027918 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:08.959061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:08.959020 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" podUID="8fbd3bf6-3386-414e-8a35-240ba641d306" containerName="main" containerID="cri-o://b5881924a6ebd8417184685813e6b0b935fd0a874f39243a58c9f991fb6b2690" gracePeriod=45 Nov 02 21:28:09.220772 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:09.220708 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" Nov 02 21:28:09.295189 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:09.295068 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fbd3bf6-3386-414e-8a35-240ba641d306-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fbd3bf6-3386-414e-8a35-240ba641d306-dshm\") pod \"8fbd3bf6-3386-414e-8a35-240ba641d306\" (UID: \"8fbd3bf6-3386-414e-8a35-240ba641d306\") " Nov 02 21:28:09.295189 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:09.295120 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fbd3bf6-3386-414e-8a35-240ba641d306-kube-api-access-lngp4\" (UniqueName: \"kubernetes.io/projected/8fbd3bf6-3386-414e-8a35-240ba641d306-kube-api-access-lngp4\") pod \"8fbd3bf6-3386-414e-8a35-240ba641d306\" (UID: \"8fbd3bf6-3386-414e-8a35-240ba641d306\") " Nov 02 21:28:09.295659 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:09.295214 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fbd3bf6-3386-414e-8a35-240ba641d306-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fbd3bf6-3386-414e-8a35-240ba641d306-tmp-dir\") pod \"8fbd3bf6-3386-414e-8a35-240ba641d306\" (UID: \"8fbd3bf6-3386-414e-8a35-240ba641d306\") " Nov 02 21:28:09.295659 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:09.295373 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fbd3bf6-3386-414e-8a35-240ba641d306-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fbd3bf6-3386-414e-8a35-240ba641d306-model-cache\") pod \"8fbd3bf6-3386-414e-8a35-240ba641d306\" (UID: \"8fbd3bf6-3386-414e-8a35-240ba641d306\") " Nov 02 21:28:09.295659 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:09.295505 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fbd3bf6-3386-414e-8a35-240ba641d306-tls-certs\" (UniqueName: \"kubernetes.io/secret/8fbd3bf6-3386-414e-8a35-240ba641d306-tls-certs\") pod \"8fbd3bf6-3386-414e-8a35-240ba641d306\" (UID: \"8fbd3bf6-3386-414e-8a35-240ba641d306\") " Nov 02 21:28:09.295659 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:09.295557 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fbd3bf6-3386-414e-8a35-240ba641d306-home\" (UniqueName: \"kubernetes.io/empty-dir/8fbd3bf6-3386-414e-8a35-240ba641d306-home\") pod \"8fbd3bf6-3386-414e-8a35-240ba641d306\" (UID: \"8fbd3bf6-3386-414e-8a35-240ba641d306\") " Nov 02 21:28:09.295659 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:09.295579 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fbd3bf6-3386-414e-8a35-240ba641d306-tmp-dir" pod "8fbd3bf6-3386-414e-8a35-240ba641d306" (UID: "8fbd3bf6-3386-414e-8a35-240ba641d306"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:28:09.295659 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:09.295598 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fbd3bf6-3386-414e-8a35-240ba641d306-model-cache" pod "8fbd3bf6-3386-414e-8a35-240ba641d306" (UID: "8fbd3bf6-3386-414e-8a35-240ba641d306"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:28:09.296037 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:09.295940 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fbd3bf6-3386-414e-8a35-240ba641d306-home" pod "8fbd3bf6-3386-414e-8a35-240ba641d306" (UID: "8fbd3bf6-3386-414e-8a35-240ba641d306"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:28:09.296096 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:09.296056 4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fbd3bf6-3386-414e-8a35-240ba641d306-tmp-dir\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:09.296096 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:09.296075 4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fbd3bf6-3386-414e-8a35-240ba641d306-model-cache\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:09.296096 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:09.296089 4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fbd3bf6-3386-414e-8a35-240ba641d306-home\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:09.297890 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:09.297858 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fbd3bf6-3386-414e-8a35-240ba641d306-tls-certs" pod "8fbd3bf6-3386-414e-8a35-240ba641d306" (UID: "8fbd3bf6-3386-414e-8a35-240ba641d306"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:28:09.298002 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:09.297946 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fbd3bf6-3386-414e-8a35-240ba641d306-dshm" pod "8fbd3bf6-3386-414e-8a35-240ba641d306" (UID: "8fbd3bf6-3386-414e-8a35-240ba641d306"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:28:09.298002 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:09.297951 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fbd3bf6-3386-414e-8a35-240ba641d306-kube-api-access-lngp4" pod "8fbd3bf6-3386-414e-8a35-240ba641d306" (UID: "8fbd3bf6-3386-414e-8a35-240ba641d306"). InnerVolumeSpecName "kube-api-access-lngp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:28:09.397320 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:09.397286 4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fbd3bf6-3386-414e-8a35-240ba641d306-tls-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:09.397320 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:09.397313 4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fbd3bf6-3386-414e-8a35-240ba641d306-dshm\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:09.397320 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:09.397322 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lngp4\" (UniqueName: \"kubernetes.io/projected/8fbd3bf6-3386-414e-8a35-240ba641d306-kube-api-access-lngp4\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:09.747052 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:09.747015 4935 generic.go:356] "Generic (PLEG): container finished" podID="8fbd3bf6-3386-414e-8a35-240ba641d306" containerID="b5881924a6ebd8417184685813e6b0b935fd0a874f39243a58c9f991fb6b2690" exitCode=0 Nov 02 21:28:09.747250 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:09.747085 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" Nov 02 21:28:09.747250 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:09.747100 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" event={"ID":"8fbd3bf6-3386-414e-8a35-240ba641d306","Type":"ContainerDied","Data":"b5881924a6ebd8417184685813e6b0b935fd0a874f39243a58c9f991fb6b2690"} Nov 02 21:28:09.747250 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:09.747174 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z" event={"ID":"8fbd3bf6-3386-414e-8a35-240ba641d306","Type":"ContainerDied","Data":"4acb6fcee5a7b02075719bc3933eeeb4724318000b849c312a28aa74828105fe"} Nov 02 21:28:09.747250 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:09.747199 4935 scope.go:122] "RemoveContainer" containerID="b5881924a6ebd8417184685813e6b0b935fd0a874f39243a58c9f991fb6b2690" Nov 02 21:28:09.757526 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:09.757500 4935 scope.go:122] "RemoveContainer" containerID="b5881924a6ebd8417184685813e6b0b935fd0a874f39243a58c9f991fb6b2690" Nov 02 21:28:09.758077 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:28:09.757997 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5881924a6ebd8417184685813e6b0b935fd0a874f39243a58c9f991fb6b2690\": container with ID starting with b5881924a6ebd8417184685813e6b0b935fd0a874f39243a58c9f991fb6b2690 not found: ID does not exist" containerID="b5881924a6ebd8417184685813e6b0b935fd0a874f39243a58c9f991fb6b2690" Nov 02 21:28:09.758077 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:09.758035 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5881924a6ebd8417184685813e6b0b935fd0a874f39243a58c9f991fb6b2690"} err="failed to get container status \"b5881924a6ebd8417184685813e6b0b935fd0a874f39243a58c9f991fb6b2690\": rpc error: code = NotFound desc = could not find container \"b5881924a6ebd8417184685813e6b0b935fd0a874f39243a58c9f991fb6b2690\": container with ID starting with b5881924a6ebd8417184685813e6b0b935fd0a874f39243a58c9f991fb6b2690 not found: ID does not exist" Nov 02 21:28:09.900337 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:09.900306 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z"] Nov 02 21:28:09.946454 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:09.946425 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/e2e-conflict-a-00e30839-kserve-985d8d67f-9b24z"] Nov 02 21:28:11.776888 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:11.776839 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fbd3bf6-3386-414e-8a35-240ba641d306" path="/var/lib/kubelet/pods/8fbd3bf6-3386-414e-8a35-240ba641d306/volumes" Nov 02 21:28:13.042792 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:13.042756 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:18.134673 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:18.134631 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:23.220069 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:23.220020 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:24.032188 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:24.032119 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:27.967186 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:27.967156 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 21:28:27.968952 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:27.968928 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 21:28:27.977762 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:27.977739 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 21:28:27.979297 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:27.979277 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 21:28:28.283669 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:28.283633 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:28.473946 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:28:28.473902 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:28:33.379186 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:33.379113 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:38.334102 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:38.334010 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:41.158842 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.158805 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85"] Nov 02 21:28:41.159281 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.159267 4935 cpu_manager.go:414] "containerMap: removing container" podUID="8fbd3bf6-3386-414e-8a35-240ba641d306" containerName="main" Nov 02 21:28:41.159327 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.159282 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fbd3bf6-3386-414e-8a35-240ba641d306" containerName="main" Nov 02 21:28:41.159394 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.159386 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fbd3bf6-3386-414e-8a35-240ba641d306" containerName="main" Nov 02 21:28:41.164076 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.164054 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.212932 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.212904 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5f765a-openshift-default-dockercfg-6vjh4\"" Nov 02 21:28:41.258195 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.258150 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.258372 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.258290 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd7972de-f06d-4db9-84ae-15dcce006b32-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.258433 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.258368 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.258433 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.258403 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.258433 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.258426 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd7972de-f06d-4db9-84ae-15dcce006b32-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.258575 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.258522 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.258621 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.258573 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd7972de-f06d-4db9-84ae-15dcce006b32-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.258703 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.258686 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.258752 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.258730 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx6fv\" (UniqueName: \"kubernetes.io/projected/cd7972de-f06d-4db9-84ae-15dcce006b32-kube-api-access-zx6fv\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.262446 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.262421 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85"] Nov 02 21:28:41.360343 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.360311 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd7972de-f06d-4db9-84ae-15dcce006b32-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.360549 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.360365 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.360549 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.360387 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.360549 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.360406 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd7972de-f06d-4db9-84ae-15dcce006b32-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.360549 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.360436 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.360549 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.360482 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd7972de-f06d-4db9-84ae-15dcce006b32-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.360822 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.360555 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.360822 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.360581 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx6fv\" (UniqueName: \"kubernetes.io/projected/cd7972de-f06d-4db9-84ae-15dcce006b32-kube-api-access-zx6fv\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.360822 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.360636 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.360822 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.360789 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.361032 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.360867 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.361032 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.360904 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.361032 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.361001 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.361531 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.361507 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd7972de-f06d-4db9-84ae-15dcce006b32-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.363159 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.363116 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.363554 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.363532 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd7972de-f06d-4db9-84ae-15dcce006b32-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.413660 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.413584 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd7972de-f06d-4db9-84ae-15dcce006b32-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.413782 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.413718 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx6fv\" (UniqueName: \"kubernetes.io/projected/cd7972de-f06d-4db9-84ae-15dcce006b32-kube-api-access-zx6fv\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.477327 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.477291 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:41.894410 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.894387 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85"] Nov 02 21:28:41.896102 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:28:41.896067 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7972de_f06d_4db9_84ae_15dcce006b32.slice/crio-72fb433d82f950132512e0a2532222217650ddb14102f359f55ab4a7f36f02d9 WatchSource:0}: Error finding container 72fb433d82f950132512e0a2532222217650ddb14102f359f55ab4a7f36f02d9: Status 404 returned error can't find the container with id 72fb433d82f950132512e0a2532222217650ddb14102f359f55ab4a7f36f02d9 Nov 02 21:28:41.898445 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.898413 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:28:41.898511 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.898478 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:28:41.898511 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.898507 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:28:41.945928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:41.945892 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" event={"ID":"cd7972de-f06d-4db9-84ae-15dcce006b32","Type":"ContainerStarted","Data":"72fb433d82f950132512e0a2532222217650ddb14102f359f55ab4a7f36f02d9"} Nov 02 21:28:42.957448 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:42.957402 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" event={"ID":"cd7972de-f06d-4db9-84ae-15dcce006b32","Type":"ContainerStarted","Data":"7ac550da95e04b036089f66afd368b6448c8a95db48ff3c791e3a4dd929262b1"} Nov 02 21:28:43.163411 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:43.163352 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" podStartSLOduration=2.1633356360000002 podStartE2EDuration="2.163335636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:28:43.112723821 +0000 UTC m=+1215.552298544" watchObservedRunningTime="2026-11-02 21:28:43.163335636 +0000 UTC m=+1215.602910359" Nov 02 21:28:43.366781 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:43.365879 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:43.478293 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:43.478259 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:43.479491 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:43.479463 4935 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 02 21:28:43.479641 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:43.479528 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" podUID="cd7972de-f06d-4db9-84ae-15dcce006b32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 02 21:28:44.478302 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:44.478266 4935 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 02 21:28:44.478808 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:44.478317 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" podUID="cd7972de-f06d-4db9-84ae-15dcce006b32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 02 21:28:45.169353 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:45.169313 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:45.478034 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:45.477934 4935 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 02 21:28:45.478034 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:45.478007 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" podUID="cd7972de-f06d-4db9-84ae-15dcce006b32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 02 21:28:46.478290 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:46.478249 4935 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 02 21:28:46.478708 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:46.478304 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" podUID="cd7972de-f06d-4db9-84ae-15dcce006b32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 02 21:28:47.478425 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:47.478393 4935 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 02 21:28:47.478827 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:47.478455 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" podUID="cd7972de-f06d-4db9-84ae-15dcce006b32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 02 21:28:48.478110 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:48.478074 4935 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 02 21:28:48.478307 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:48.478170 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" podUID="cd7972de-f06d-4db9-84ae-15dcce006b32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 02 21:28:48.524324 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:48.524249 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:49.482747 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:49.482709 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:49.483162 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:49.483117 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:49.483786 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:49.483767 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:28:52.932398 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:52.932318 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:53.616861 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:53.616826 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:28:58.471523 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:28:58.471483 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:28:59.011621 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:28:59.011585 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:04.040315 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:04.040269 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:08.188484 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.188448 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k"] Nov 02 21:29:08.192586 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.192564 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.278342 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.278302 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.278528 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.278441 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.278528 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.278479 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.278528 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.278514 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.278721 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.278545 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.278721 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.278680 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gqwq\" (UniqueName: \"kubernetes.io/projected/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-kube-api-access-7gqwq\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.278829 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.278738 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.278829 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.278777 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.278829 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.278812 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.376715 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.376678 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k"] Nov 02 21:29:08.380593 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.380558 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gqwq\" (UniqueName: \"kubernetes.io/projected/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-kube-api-access-7gqwq\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.380740 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.380634 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.380740 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.380678 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.380740 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.380710 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.380924 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.380751 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.380924 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.380823 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.380924 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.380846 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.380924 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.380878 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.380924 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.380909 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.381333 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.381307 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-workload-certs\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.381406 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.381351 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-workload-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.381705 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.381683 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istio-data\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.381781 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.381701 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-credential-socket\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.382737 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.382712 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istiod-ca-cert\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.384488 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.384463 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istio-envoy\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.384922 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.384893 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istio-podinfo\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.433921 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.433894 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gqwq\" (UniqueName: \"kubernetes.io/projected/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-kube-api-access-7gqwq\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.434502 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.434484 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istio-token\") pod \"e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.506331 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.506267 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:08.924193 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.924151 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k"] Nov 02 21:29:08.925718 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:29:08.925689 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f86cdaf_d4dd_4281_ad28_f2750dab88a9.slice/crio-6d9b96d475bef3096f7b22961b24c261d030ef5059b8abbb413bd69304c99372 WatchSource:0}: Error finding container 6d9b96d475bef3096f7b22961b24c261d030ef5059b8abbb413bd69304c99372: Status 404 returned error can't find the container with id 6d9b96d475bef3096f7b22961b24c261d030ef5059b8abbb413bd69304c99372 Nov 02 21:29:08.928116 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.928082 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:29:08.928215 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.928167 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:29:08.928215 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:08.928202 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:29:09.059399 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:09.059364 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:09.167159 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:09.167105 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" event={"ID":"7f86cdaf-d4dd-4281-ad28-f2750dab88a9","Type":"ContainerStarted","Data":"ee6b4830c628e6e019b85b522551dbda2238ce2a6a6cb461050d44df07f4a0be"} Nov 02 21:29:09.167342 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:09.167187 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" event={"ID":"7f86cdaf-d4dd-4281-ad28-f2750dab88a9","Type":"ContainerStarted","Data":"6d9b96d475bef3096f7b22961b24c261d030ef5059b8abbb413bd69304c99372"} Nov 02 21:29:09.302434 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:09.302384 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" podStartSLOduration=1.302369917 podStartE2EDuration="1.302369917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:09.299332662 +0000 UTC m=+1241.738907394" watchObservedRunningTime="2026-11-02 21:29:09.302369917 +0000 UTC m=+1241.741944638" Nov 02 21:29:09.506600 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:09.506567 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:09.508061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:09.508038 4935 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 21:29:09.508184 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:09.508099 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" podUID="7f86cdaf-d4dd-4281-ad28-f2750dab88a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 21:29:10.507496 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:10.507458 4935 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 21:29:10.507972 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:10.507531 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" podUID="7f86cdaf-d4dd-4281-ad28-f2750dab88a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 21:29:11.507509 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:11.507472 4935 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 21:29:11.507915 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:11.507531 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" podUID="7f86cdaf-d4dd-4281-ad28-f2750dab88a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 21:29:12.507425 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:12.507389 4935 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 21:29:12.507581 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:12.507447 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" podUID="7f86cdaf-d4dd-4281-ad28-f2750dab88a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 21:29:13.507049 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:13.507011 4935 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 21:29:13.507240 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:13.507080 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" podUID="7f86cdaf-d4dd-4281-ad28-f2750dab88a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 21:29:14.362895 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:14.362854 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:14.507044 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:14.507002 4935 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 21:29:14.507258 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:14.507065 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" podUID="7f86cdaf-d4dd-4281-ad28-f2750dab88a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 21:29:15.507571 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:15.507534 4935 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 21:29:15.508174 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:15.508120 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" podUID="7f86cdaf-d4dd-4281-ad28-f2750dab88a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 21:29:17.513244 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:17.513198 4935 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:29:17.513745 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:17.513259 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" podUID="7f86cdaf-d4dd-4281-ad28-f2750dab88a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:29:18.246503 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:18.246469 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:18.247344 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:18.247319 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:18.248088 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:18.248061 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:29:19.723723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:19.723686 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:24.362568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:24.362522 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:24.753026 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:24.752918 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:25.971963 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:25.971929 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt"] Nov 02 21:29:25.975913 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:25.975894 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" Nov 02 21:29:26.028917 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:26.028875 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt"] Nov 02 21:29:26.071296 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:26.071261 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d6cb38\"/\"distinct-2-w4-d6cb38-kserve-self-signed-certs\"" Nov 02 21:29:26.079685 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:26.079652 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095af893-e150-43b9-b479-263a270d05c4-home\") pod \"distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt\" (UID: \"095af893-e150-43b9-b479-263a270d05c4\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" Nov 02 21:29:26.079834 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:26.079706 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095af893-e150-43b9-b479-263a270d05c4-dshm\") pod \"distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt\" (UID: \"095af893-e150-43b9-b479-263a270d05c4\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" Nov 02 21:29:26.079834 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:26.079804 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwh4q\" (UniqueName: \"kubernetes.io/projected/095af893-e150-43b9-b479-263a270d05c4-kube-api-access-bwh4q\") pod \"distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt\" (UID: \"095af893-e150-43b9-b479-263a270d05c4\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" Nov 02 21:29:26.079954 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:26.079836 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095af893-e150-43b9-b479-263a270d05c4-tmp-dir\") pod \"distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt\" (UID: \"095af893-e150-43b9-b479-263a270d05c4\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" Nov 02 21:29:26.079954 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:26.079937 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095af893-e150-43b9-b479-263a270d05c4-model-cache\") pod \"distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt\" (UID: \"095af893-e150-43b9-b479-263a270d05c4\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" Nov 02 21:29:26.080060 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:26.079975 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095af893-e150-43b9-b479-263a270d05c4-tls-certs\") pod \"distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt\" (UID: \"095af893-e150-43b9-b479-263a270d05c4\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" Nov 02 21:29:26.149259 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:26.149222 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:26.181453 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:26.181414 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095af893-e150-43b9-b479-263a270d05c4-home\") pod \"distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt\" (UID: \"095af893-e150-43b9-b479-263a270d05c4\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" Nov 02 21:29:26.181619 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:26.181487 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095af893-e150-43b9-b479-263a270d05c4-dshm\") pod \"distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt\" (UID: \"095af893-e150-43b9-b479-263a270d05c4\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" Nov 02 21:29:26.181619 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:26.181559 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwh4q\" (UniqueName: \"kubernetes.io/projected/095af893-e150-43b9-b479-263a270d05c4-kube-api-access-bwh4q\") pod \"distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt\" (UID: \"095af893-e150-43b9-b479-263a270d05c4\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" Nov 02 21:29:26.181619 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:26.181596 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095af893-e150-43b9-b479-263a270d05c4-tmp-dir\") pod \"distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt\" (UID: \"095af893-e150-43b9-b479-263a270d05c4\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" Nov 02 21:29:26.181799 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:26.181663 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095af893-e150-43b9-b479-263a270d05c4-model-cache\") pod \"distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt\" (UID: \"095af893-e150-43b9-b479-263a270d05c4\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" Nov 02 21:29:26.181799 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:26.181706 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095af893-e150-43b9-b479-263a270d05c4-tls-certs\") pod \"distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt\" (UID: \"095af893-e150-43b9-b479-263a270d05c4\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" Nov 02 21:29:26.181940 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:26.181912 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095af893-e150-43b9-b479-263a270d05c4-home\") pod \"distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt\" (UID: \"095af893-e150-43b9-b479-263a270d05c4\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" Nov 02 21:29:26.182092 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:26.182071 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095af893-e150-43b9-b479-263a270d05c4-tmp-dir\") pod \"distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt\" (UID: \"095af893-e150-43b9-b479-263a270d05c4\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" Nov 02 21:29:26.182211 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:26.182189 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095af893-e150-43b9-b479-263a270d05c4-model-cache\") pod \"distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt\" (UID: \"095af893-e150-43b9-b479-263a270d05c4\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" Nov 02 21:29:26.184236 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:26.184204 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095af893-e150-43b9-b479-263a270d05c4-dshm\") pod \"distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt\" (UID: \"095af893-e150-43b9-b479-263a270d05c4\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" Nov 02 21:29:26.184338 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:26.184244 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095af893-e150-43b9-b479-263a270d05c4-tls-certs\") pod \"distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt\" (UID: \"095af893-e150-43b9-b479-263a270d05c4\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" Nov 02 21:29:26.235826 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:26.235752 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwh4q\" (UniqueName: \"kubernetes.io/projected/095af893-e150-43b9-b479-263a270d05c4-kube-api-access-bwh4q\") pod \"distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt\" (UID: \"095af893-e150-43b9-b479-263a270d05c4\") " pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" Nov 02 21:29:26.288540 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:26.288489 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" Nov 02 21:29:26.680204 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:26.680175 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt"] Nov 02 21:29:26.682425 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:29:26.682395 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod095af893_e150_43b9_b479_263a270d05c4.slice/crio-834c15c7e2d55eb910c78444011a9d40e3524a64271d25455e569f01b7b36b43 WatchSource:0}: Error finding container 834c15c7e2d55eb910c78444011a9d40e3524a64271d25455e569f01b7b36b43: Status 404 returned error can't find the container with id 834c15c7e2d55eb910c78444011a9d40e3524a64271d25455e569f01b7b36b43 Nov 02 21:29:27.307746 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:27.307688 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" event={"ID":"095af893-e150-43b9-b479-263a270d05c4","Type":"ContainerStarted","Data":"c713f4ce3c43a590bdeeb5ba34c2e4f9516edcb8b56f22467e5c93a21d04b3b3"} Nov 02 21:29:27.308334 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:27.308302 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" event={"ID":"095af893-e150-43b9-b479-263a270d05c4","Type":"ContainerStarted","Data":"834c15c7e2d55eb910c78444011a9d40e3524a64271d25455e569f01b7b36b43"} Nov 02 21:29:27.416142 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:27.416078 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" podStartSLOduration=2.416060725 podStartE2EDuration="2.416060725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:27.413967544 +0000 UTC m=+1259.853542267" watchObservedRunningTime="2026-11-02 21:29:27.416060725 +0000 UTC m=+1259.855635448" Nov 02 21:29:28.288633 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:28.288600 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" Nov 02 21:29:28.472150 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:29:28.472093 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:29:29.838454 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:29.838415 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:34.934730 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:34.934686 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:36.288638 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:36.288598 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" Nov 02 21:29:36.301462 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:36.301430 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" Nov 02 21:29:36.397150 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:36.397098 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" Nov 02 21:29:39.952397 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:39.952352 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:41.616562 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.616525 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6"] Nov 02 21:29:41.621629 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.621607 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.664693 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.664664 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn"] Nov 02 21:29:41.668486 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.668466 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:41.674502 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.674475 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6"] Nov 02 21:29:41.708884 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.708852 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qwfg\" (UniqueName: \"kubernetes.io/projected/83fbff2a-a71b-40bd-a636-58c762e22aee-kube-api-access-9qwfg\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:41.709042 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.708903 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83fbff2a-a71b-40bd-a636-58c762e22aee-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:41.709042 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.708971 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.709205 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.709037 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.709205 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.709062 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.709314 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.709214 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49d7528a-f6fa-49f0-97a7-461066310602-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.709314 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.709249 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49d7528a-f6fa-49f0-97a7-461066310602-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.709314 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.709282 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:41.709498 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.709464 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:41.709557 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.709513 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49d7528a-f6fa-49f0-97a7-461066310602-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.709620 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.709600 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:41.709680 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.709654 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83fbff2a-a71b-40bd-a636-58c762e22aee-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:41.709731 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.709675 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:41.709731 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.709696 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.709840 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.709728 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:41.709840 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.709759 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hdkw\" (UniqueName: \"kubernetes.io/projected/49d7528a-f6fa-49f0-97a7-461066310602-kube-api-access-7hdkw\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.709840 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.709822 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83fbff2a-a71b-40bd-a636-58c762e22aee-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:41.709991 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.709844 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.731520 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.731486 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn"] Nov 02 21:29:41.811053 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.811018 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.811258 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.811059 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.811258 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.811124 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49d7528a-f6fa-49f0-97a7-461066310602-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.811258 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.811175 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49d7528a-f6fa-49f0-97a7-461066310602-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.811258 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.811201 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:41.811480 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.811323 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:41.811480 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.811350 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49d7528a-f6fa-49f0-97a7-461066310602-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.811480 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.811399 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:41.811480 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.811435 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83fbff2a-a71b-40bd-a636-58c762e22aee-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:41.811742 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.811469 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:41.811742 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.811500 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.811742 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.811518 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.811742 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.811540 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.811742 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.811614 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:41.811742 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.811644 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hdkw\" (UniqueName: \"kubernetes.io/projected/49d7528a-f6fa-49f0-97a7-461066310602-kube-api-access-7hdkw\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.811742 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.811698 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:41.812180 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.811793 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83fbff2a-a71b-40bd-a636-58c762e22aee-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:41.812180 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.811846 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.812180 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.811891 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-credential-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:41.812180 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.811917 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qwfg\" (UniqueName: \"kubernetes.io/projected/83fbff2a-a71b-40bd-a636-58c762e22aee-kube-api-access-9qwfg\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:41.812180 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.811989 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83fbff2a-a71b-40bd-a636-58c762e22aee-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:41.812180 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.812028 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.812180 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.812021 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.812568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.812191 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-workload-certs\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:41.812568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.812500 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-workload-socket\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:41.812674 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.812652 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49d7528a-f6fa-49f0-97a7-461066310602-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.812813 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.812765 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-istio-data\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.813115 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.813091 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83fbff2a-a71b-40bd-a636-58c762e22aee-istiod-ca-cert\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:41.814065 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.814041 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49d7528a-f6fa-49f0-97a7-461066310602-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.814221 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.814204 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.815183 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.815159 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-istio-envoy\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:41.815302 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.815285 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83fbff2a-a71b-40bd-a636-58c762e22aee-istio-podinfo\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:41.864709 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.864680 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83fbff2a-a71b-40bd-a636-58c762e22aee-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:41.864840 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.864682 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49d7528a-f6fa-49f0-97a7-461066310602-istio-token\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.864840 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.864743 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qwfg\" (UniqueName: \"kubernetes.io/projected/83fbff2a-a71b-40bd-a636-58c762e22aee-kube-api-access-9qwfg\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:41.910898 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.910834 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hdkw\" (UniqueName: \"kubernetes.io/projected/49d7528a-f6fa-49f0-97a7-461066310602-kube-api-access-7hdkw\") pod \"e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.935764 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.935734 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:41.982348 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:41.982319 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:42.356599 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:42.356559 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6"] Nov 02 21:29:42.394009 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:29:42.393975 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d7528a_f6fa_49f0_97a7_461066310602.slice/crio-1c076342f3c5e04f56f41e333639a9d71baaff67a50da68c90d2a15dd2b85d5d WatchSource:0}: Error finding container 1c076342f3c5e04f56f41e333639a9d71baaff67a50da68c90d2a15dd2b85d5d: Status 404 returned error can't find the container with id 1c076342f3c5e04f56f41e333639a9d71baaff67a50da68c90d2a15dd2b85d5d Nov 02 21:29:42.396791 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:42.396752 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:29:42.396900 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:42.396858 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:29:42.396960 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:42.396914 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:29:42.410343 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:42.410311 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn"] Nov 02 21:29:42.416004 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:29:42.415968 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83fbff2a_a71b_40bd_a636_58c762e22aee.slice/crio-f38bb92b6e9b71c309d4530dd17275711ff41f4d3529d5979bd8d71974af843d WatchSource:0}: Error finding container f38bb92b6e9b71c309d4530dd17275711ff41f4d3529d5979bd8d71974af843d: Status 404 returned error can't find the container with id f38bb92b6e9b71c309d4530dd17275711ff41f4d3529d5979bd8d71974af843d Nov 02 21:29:42.418624 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:42.418594 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:29:42.418716 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:42.418658 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:29:42.418716 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:42.418689 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:29:42.427021 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:42.426992 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" event={"ID":"83fbff2a-a71b-40bd-a636-58c762e22aee","Type":"ContainerStarted","Data":"f38bb92b6e9b71c309d4530dd17275711ff41f4d3529d5979bd8d71974af843d"} Nov 02 21:29:42.428396 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:42.428365 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" event={"ID":"49d7528a-f6fa-49f0-97a7-461066310602","Type":"ContainerStarted","Data":"1c076342f3c5e04f56f41e333639a9d71baaff67a50da68c90d2a15dd2b85d5d"} Nov 02 21:29:43.473156 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:43.462320 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" event={"ID":"49d7528a-f6fa-49f0-97a7-461066310602","Type":"ContainerStarted","Data":"94e12b06d4c1e7e110bbdaf8d16b44cc115fdc4e3f4c98b007f1254bd90e3900"} Nov 02 21:29:43.473156 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:43.465649 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" event={"ID":"83fbff2a-a71b-40bd-a636-58c762e22aee","Type":"ContainerStarted","Data":"361105a3c6d5b601fdc89e15c2faee4a4a856032b88b3452ce8c32b8d0b6982f"} Nov 02 21:29:43.670155 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:43.669201 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" podStartSLOduration=2.669180029 podStartE2EDuration="2.669180029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:43.620399491 +0000 UTC m=+1276.059974213" watchObservedRunningTime="2026-11-02 21:29:43.669180029 +0000 UTC m=+1276.108754752" Nov 02 21:29:43.766212 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:43.761399 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" podStartSLOduration=2.7613779750000003 podStartE2EDuration="2.761377975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:43.739713733 +0000 UTC m=+1276.179288483" watchObservedRunningTime="2026-11-02 21:29:43.761377975 +0000 UTC m=+1276.200952699" Nov 02 21:29:43.962155 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:43.951543 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:43.972156 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:43.970343 4935 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 21:29:43.972156 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:43.970430 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" podUID="49d7528a-f6fa-49f0-97a7-461066310602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 21:29:43.986260 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:43.986215 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:43.987841 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:43.987811 4935 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 21:29:43.987926 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:43.987876 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" podUID="83fbff2a-a71b-40bd-a636-58c762e22aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 21:29:44.942262 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:44.941587 4935 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 21:29:44.942262 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:44.941651 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" podUID="49d7528a-f6fa-49f0-97a7-461066310602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 21:29:44.983082 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:44.983042 4935 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 21:29:44.983300 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:44.983095 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" podUID="83fbff2a-a71b-40bd-a636-58c762e22aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 21:29:45.201183 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:45.201024 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:45.943163 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:45.942686 4935 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 21:29:45.943163 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:45.942753 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" podUID="49d7528a-f6fa-49f0-97a7-461066310602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 21:29:45.983426 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:45.983396 4935 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 21:29:45.983648 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:45.983627 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" podUID="83fbff2a-a71b-40bd-a636-58c762e22aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 21:29:46.938154 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:46.936593 4935 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 21:29:46.938154 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:46.936672 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" podUID="49d7528a-f6fa-49f0-97a7-461066310602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 21:29:46.983317 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:46.983274 4935 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 21:29:46.983837 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:46.983805 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" podUID="83fbff2a-a71b-40bd-a636-58c762e22aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 21:29:47.947158 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:47.946511 4935 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 21:29:47.947158 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:47.946589 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" podUID="49d7528a-f6fa-49f0-97a7-461066310602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 21:29:47.983412 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:47.983369 4935 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 21:29:47.983811 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:47.983421 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" podUID="83fbff2a-a71b-40bd-a636-58c762e22aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 21:29:48.937025 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:48.936981 4935 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 21:29:48.937344 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:48.937308 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" podUID="49d7528a-f6fa-49f0-97a7-461066310602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 21:29:48.983365 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:48.983326 4935 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 21:29:48.983529 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:48.983391 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" podUID="83fbff2a-a71b-40bd-a636-58c762e22aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 21:29:49.940486 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:49.940455 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:49.940835 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:49.940812 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:49.941726 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:49.941703 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:29:49.986783 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:49.986760 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:49.987237 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:49.987105 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:49.987671 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:49.987654 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:29:50.645961 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:50.645925 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:56.321561 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:56.321523 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:56.403278 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:56.403229 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" podUID="aab5db02-4890-45e1-bcf8-1c43c335a393" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:29:56.432290 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:56.432244 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" podUID="095af893-e150-43b9-b479-263a270d05c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:29:57.576829 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:57.576790 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:58.160098 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:58.160037 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:29:58.493763 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:29:58.493678 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:29:58.757119 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:29:58.757017 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:00.168231 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:00.168191 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894250-f7dhw"] Nov 02 21:30:00.172756 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:00.172723 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-f7dhw" Nov 02 21:30:00.221522 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:00.221491 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\"" Nov 02 21:30:00.245415 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:00.245380 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42cfaf5c-edee-4b8c-b8e8-05bfda270667-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894250-f7dhw\" (UID: \"42cfaf5c-edee-4b8c-b8e8-05bfda270667\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-f7dhw" Nov 02 21:30:00.245564 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:00.245532 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr7n7\" (UniqueName: \"kubernetes.io/projected/42cfaf5c-edee-4b8c-b8e8-05bfda270667-kube-api-access-mr7n7\") pod \"maas-api-key-cleanup-29894250-f7dhw\" (UID: \"42cfaf5c-edee-4b8c-b8e8-05bfda270667\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-f7dhw" Nov 02 21:30:00.272851 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:00.272814 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894250-f7dhw"] Nov 02 21:30:00.347243 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:00.347209 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42cfaf5c-edee-4b8c-b8e8-05bfda270667-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894250-f7dhw\" (UID: \"42cfaf5c-edee-4b8c-b8e8-05bfda270667\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-f7dhw" Nov 02 21:30:00.347431 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:00.347373 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr7n7\" (UniqueName: \"kubernetes.io/projected/42cfaf5c-edee-4b8c-b8e8-05bfda270667-kube-api-access-mr7n7\") pod \"maas-api-key-cleanup-29894250-f7dhw\" (UID: \"42cfaf5c-edee-4b8c-b8e8-05bfda270667\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-f7dhw" Nov 02 21:30:00.348148 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:00.348110 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42cfaf5c-edee-4b8c-b8e8-05bfda270667-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894250-f7dhw\" (UID: \"42cfaf5c-edee-4b8c-b8e8-05bfda270667\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-f7dhw" Nov 02 21:30:00.401878 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:00.401848 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr7n7\" (UniqueName: \"kubernetes.io/projected/42cfaf5c-edee-4b8c-b8e8-05bfda270667-kube-api-access-mr7n7\") pod \"maas-api-key-cleanup-29894250-f7dhw\" (UID: \"42cfaf5c-edee-4b8c-b8e8-05bfda270667\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-f7dhw" Nov 02 21:30:00.487066 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:00.486987 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-f7dhw" Nov 02 21:30:00.861444 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:00.861409 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894250-f7dhw"] Nov 02 21:30:00.863904 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:30:00.863874 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42cfaf5c_edee_4b8c_b8e8_05bfda270667.slice/crio-54def84541a94c61712c58615921028b6194151403f1b907cfa915d2d3798011 WatchSource:0}: Error finding container 54def84541a94c61712c58615921028b6194151403f1b907cfa915d2d3798011: Status 404 returned error can't find the container with id 54def84541a94c61712c58615921028b6194151403f1b907cfa915d2d3798011 Nov 02 21:30:00.866682 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:00.866652 4935 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:30:01.669098 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:01.669064 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-f7dhw" event={"ID":"42cfaf5c-edee-4b8c-b8e8-05bfda270667","Type":"ContainerStarted","Data":"54def84541a94c61712c58615921028b6194151403f1b907cfa915d2d3798011"} Nov 02 21:30:01.743800 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:01.743762 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:02.719958 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:02.719912 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:02.959525 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:02.959489 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:03.179098 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:03.179065 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:03.395111 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:03.395080 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:04.280200 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:04.280169 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:04.498210 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:04.498175 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:05.857180 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:05.857116 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:06.064284 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:06.064246 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:07.411773 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:07.411736 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:10.277220 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:10.276963 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:10.479634 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:10.479602 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:10.734073 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:10.733982 4935 generic.go:356] "Generic (PLEG): container finished" podID="42cfaf5c-edee-4b8c-b8e8-05bfda270667" containerID="5a946b61ef7764edf59329fa01fc47910a53efcf984ba6c00786b6b88e8571e6" exitCode=60 Nov 02 21:30:10.734250 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:10.734063 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-f7dhw" event={"ID":"42cfaf5c-edee-4b8c-b8e8-05bfda270667","Type":"ContainerDied","Data":"5a946b61ef7764edf59329fa01fc47910a53efcf984ba6c00786b6b88e8571e6"} Nov 02 21:30:10.734375 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:10.734356 4935 scope.go:122] "RemoveContainer" containerID="5a946b61ef7764edf59329fa01fc47910a53efcf984ba6c00786b6b88e8571e6" Nov 02 21:30:11.564116 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:11.564083 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f750d0c-r5w2v"] Nov 02 21:30:11.568266 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:11.568241 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f750d0c-r5w2v" Nov 02 21:30:11.616442 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:11.616413 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:30:11.712160 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:11.711612 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f750d0c-r5w2v\" (UID: \"696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f750d0c-r5w2v" Nov 02 21:30:11.712160 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:11.711783 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv7n6\" (UniqueName: \"kubernetes.io/projected/696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b-kube-api-access-nv7n6\") pod \"maas-api-revoke-sub-5f750d0c-r5w2v\" (UID: \"696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f750d0c-r5w2v" Nov 02 21:30:11.712160 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:11.711888 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f750d0c-r5w2v\" (UID: \"696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f750d0c-r5w2v" Nov 02 21:30:11.733373 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:11.730340 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f750d0c-r5w2v"] Nov 02 21:30:11.769701 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:11.765933 4935 generic.go:356] "Generic (PLEG): container finished" podID="42cfaf5c-edee-4b8c-b8e8-05bfda270667" containerID="9f4d08cbb0526f27d33f6dedd76ba575ecbed9e8883d251b0f4eb4c3ed22db7f" exitCode=60 Nov 02 21:30:11.769701 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:11.765966 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-f7dhw" event={"ID":"42cfaf5c-edee-4b8c-b8e8-05bfda270667","Type":"ContainerDied","Data":"9f4d08cbb0526f27d33f6dedd76ba575ecbed9e8883d251b0f4eb4c3ed22db7f"} Nov 02 21:30:11.769701 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:11.766040 4935 scope.go:122] "RemoveContainer" containerID="5a946b61ef7764edf59329fa01fc47910a53efcf984ba6c00786b6b88e8571e6" Nov 02 21:30:11.776307 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:11.776282 4935 scope.go:122] "RemoveContainer" containerID="9f4d08cbb0526f27d33f6dedd76ba575ecbed9e8883d251b0f4eb4c3ed22db7f" Nov 02 21:30:11.777207 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:30:11.777176 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894250-f7dhw_odh-ai-gateway-infra(42cfaf5c-edee-4b8c-b8e8-05bfda270667)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-f7dhw" podUID="42cfaf5c-edee-4b8c-b8e8-05bfda270667" Nov 02 21:30:11.813471 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:11.813432 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv7n6\" (UniqueName: \"kubernetes.io/projected/696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b-kube-api-access-nv7n6\") pod \"maas-api-revoke-sub-5f750d0c-r5w2v\" (UID: \"696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f750d0c-r5w2v" Nov 02 21:30:11.813646 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:11.813558 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f750d0c-r5w2v\" (UID: \"696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f750d0c-r5w2v" Nov 02 21:30:11.813646 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:11.813636 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f750d0c-r5w2v\" (UID: \"696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f750d0c-r5w2v" Nov 02 21:30:11.814395 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:11.814329 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f750d0c-r5w2v\" (UID: \"696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f750d0c-r5w2v" Nov 02 21:30:11.816027 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:11.816000 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f750d0c-r5w2v\" (UID: \"696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f750d0c-r5w2v" Nov 02 21:30:11.867007 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:11.866974 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv7n6\" (UniqueName: \"kubernetes.io/projected/696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b-kube-api-access-nv7n6\") pod \"maas-api-revoke-sub-5f750d0c-r5w2v\" (UID: \"696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f750d0c-r5w2v" Nov 02 21:30:11.882316 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:11.882290 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f750d0c-r5w2v" Nov 02 21:30:12.253213 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:12.253174 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f750d0c-r5w2v"] Nov 02 21:30:12.254792 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:30:12.254764 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod696dc3c6_32aa_4eb3_984e_cae3a4dc7b3b.slice/crio-ebf9dcbb2239b1caaa4ec330809a4aa8cae1c457356b08c0480ab3bf92e22d9c WatchSource:0}: Error finding container ebf9dcbb2239b1caaa4ec330809a4aa8cae1c457356b08c0480ab3bf92e22d9c: Status 404 returned error can't find the container with id ebf9dcbb2239b1caaa4ec330809a4aa8cae1c457356b08c0480ab3bf92e22d9c Nov 02 21:30:12.727010 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:12.726965 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:12.774327 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:12.774290 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f750d0c-r5w2v" event={"ID":"696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b","Type":"ContainerStarted","Data":"0a0879756b1082f42a72a10d26f321fa19942903a230fe152906f2d386058fc0"} Nov 02 21:30:12.774489 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:12.774384 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f750d0c-r5w2v" event={"ID":"696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b","Type":"ContainerStarted","Data":"ebf9dcbb2239b1caaa4ec330809a4aa8cae1c457356b08c0480ab3bf92e22d9c"} Nov 02 21:30:12.776359 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:12.776338 4935 scope.go:122] "RemoveContainer" containerID="9f4d08cbb0526f27d33f6dedd76ba575ecbed9e8883d251b0f4eb4c3ed22db7f" Nov 02 21:30:12.776623 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:30:12.776604 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894250-f7dhw_odh-ai-gateway-infra(42cfaf5c-edee-4b8c-b8e8-05bfda270667)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-f7dhw" podUID="42cfaf5c-edee-4b8c-b8e8-05bfda270667" Nov 02 21:30:12.880522 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:12.880405 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f750d0c-r5w2v" podStartSLOduration=1.880390517 podStartE2EDuration="1.880390517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:12.879414321 +0000 UTC m=+1305.318989044" watchObservedRunningTime="2026-11-02 21:30:12.880390517 +0000 UTC m=+1305.319965238" Nov 02 21:30:13.783648 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:13.783614 4935 generic.go:356] "Generic (PLEG): container finished" podID="696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b" containerID="0a0879756b1082f42a72a10d26f321fa19942903a230fe152906f2d386058fc0" exitCode=0 Nov 02 21:30:13.784076 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:13.783699 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f750d0c-r5w2v" event={"ID":"696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b","Type":"ContainerDied","Data":"0a0879756b1082f42a72a10d26f321fa19942903a230fe152906f2d386058fc0"} Nov 02 21:30:14.937367 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:14.937341 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f750d0c-r5w2v" Nov 02 21:30:14.963658 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:14.963627 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b-maas-api-openshift-ca\") pod \"696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b\" (UID: \"696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b\") " Nov 02 21:30:14.963801 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:14.963721 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b-maas-api-xks-ca\") pod \"696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b\" (UID: \"696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b\") " Nov 02 21:30:14.963918 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:14.963904 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b-kube-api-access-nv7n6\" (UniqueName: \"kubernetes.io/projected/696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b-kube-api-access-nv7n6\") pod \"696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b\" (UID: \"696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b\") " Nov 02 21:30:14.963988 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:14.963935 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b-maas-api-openshift-ca" pod "696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b" (UID: "696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:30:14.964290 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:14.964273 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b-maas-api-openshift-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:14.966351 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:14.966329 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b-maas-api-xks-ca" pod "696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b" (UID: "696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:30:14.966351 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:14.966341 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b-kube-api-access-nv7n6" pod "696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b" (UID: "696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b"). InnerVolumeSpecName "kube-api-access-nv7n6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:30:15.065148 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:15.065054 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b-maas-api-xks-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:15.065148 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:15.065089 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv7n6\" (UniqueName: \"kubernetes.io/projected/696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b-kube-api-access-nv7n6\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:15.797163 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:15.797122 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f750d0c-r5w2v" Nov 02 21:30:15.797306 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:15.797124 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f750d0c-r5w2v" event={"ID":"696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b","Type":"ContainerDied","Data":"ebf9dcbb2239b1caaa4ec330809a4aa8cae1c457356b08c0480ab3bf92e22d9c"} Nov 02 21:30:15.797306 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:15.797258 4935 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebf9dcbb2239b1caaa4ec330809a4aa8cae1c457356b08c0480ab3bf92e22d9c" Nov 02 21:30:17.843154 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:17.843050 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:22.379160 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.379043 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:22.752550 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.752473 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r"] Nov 02 21:30:22.753005 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.752992 4935 cpu_manager.go:414] "containerMap: removing container" podUID="696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b" containerName="revoke-keys" Nov 02 21:30:22.753047 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.753006 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b" containerName="revoke-keys" Nov 02 21:30:22.753099 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.753090 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b" containerName="revoke-keys" Nov 02 21:30:22.756598 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.756576 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:22.798540 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.798509 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq"] Nov 02 21:30:22.802438 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.802417 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:22.845710 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.845677 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r"] Nov 02 21:30:22.865297 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.865263 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:22.865471 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.865305 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:22.865471 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.865326 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:22.865471 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.865348 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swbzk\" (UniqueName: \"kubernetes.io/projected/ee0a0727-6bb6-4f61-82cd-80e292cbf912-kube-api-access-swbzk\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:22.865471 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.865379 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:22.865471 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.865394 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:22.865471 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.865424 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a290387e-9211-4298-a795-ac80d10f5ae9-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:22.865471 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.865462 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:22.865860 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.865510 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:22.865860 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.865530 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:22.865860 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.865545 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzn7x\" (UniqueName: \"kubernetes.io/projected/a290387e-9211-4298-a795-ac80d10f5ae9-kube-api-access-mzn7x\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:22.865860 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.865562 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:22.865860 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.865586 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:22.865860 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.865606 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:22.865860 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.865695 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a290387e-9211-4298-a795-ac80d10f5ae9-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:22.865860 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.865739 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:22.865860 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.865759 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:22.865860 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.865794 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a290387e-9211-4298-a795-ac80d10f5ae9-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:22.893256 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.893219 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq"] Nov 02 21:30:22.967373 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.967331 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:22.967556 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.967383 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:22.967556 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.967414 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzn7x\" (UniqueName: \"kubernetes.io/projected/a290387e-9211-4298-a795-ac80d10f5ae9-kube-api-access-mzn7x\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:22.967556 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.967443 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:22.967556 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.967482 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:22.967556 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.967504 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:22.967840 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.967604 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a290387e-9211-4298-a795-ac80d10f5ae9-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:22.967840 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.967660 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:22.967840 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.967683 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:22.967840 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.967731 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a290387e-9211-4298-a795-ac80d10f5ae9-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:22.967840 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.967764 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:22.967840 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.967796 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:22.967840 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.967821 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:22.967840 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.967834 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:22.968247 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.967857 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swbzk\" (UniqueName: \"kubernetes.io/projected/ee0a0727-6bb6-4f61-82cd-80e292cbf912-kube-api-access-swbzk\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:22.968247 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.967892 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:22.968247 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.968081 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:22.968247 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.967899 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:22.968247 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.968119 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:22.968247 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.968188 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a290387e-9211-4298-a795-ac80d10f5ae9-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:22.968247 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.968221 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:22.968247 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.968248 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:22.968653 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.968482 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:22.968653 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.968644 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:22.969016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.968914 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:22.969765 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.969737 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a290387e-9211-4298-a795-ac80d10f5ae9-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:22.970119 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.969920 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:22.970226 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.970183 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:22.970354 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.970326 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:22.971020 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.970993 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:22.972169 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.972088 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:22.972867 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:22.972841 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a290387e-9211-4298-a795-ac80d10f5ae9-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:23.021430 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:23.021401 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a290387e-9211-4298-a795-ac80d10f5ae9-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:23.021629 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:23.021605 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzn7x\" (UniqueName: \"kubernetes.io/projected/a290387e-9211-4298-a795-ac80d10f5ae9-kube-api-access-mzn7x\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:23.021925 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:23.021907 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swbzk\" (UniqueName: \"kubernetes.io/projected/ee0a0727-6bb6-4f61-82cd-80e292cbf912-kube-api-access-swbzk\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:23.022057 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:23.022035 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:23.070767 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:23.070732 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:23.116998 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:23.116862 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:23.345226 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:23.343044 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:23.507493 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:23.507461 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r"] Nov 02 21:30:23.511793 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:30:23.511762 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda290387e_9211_4298_a795_ac80d10f5ae9.slice/crio-a45533ac57c91fa4996e1d335bcd0599b3c8ba8cd4cbc43496dc0483daa1a6c8 WatchSource:0}: Error finding container a45533ac57c91fa4996e1d335bcd0599b3c8ba8cd4cbc43496dc0483daa1a6c8: Status 404 returned error can't find the container with id a45533ac57c91fa4996e1d335bcd0599b3c8ba8cd4cbc43496dc0483daa1a6c8 Nov 02 21:30:23.514271 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:23.514225 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:30:23.514375 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:23.514321 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:30:23.514430 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:23.514373 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:30:23.556879 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:23.556776 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq"] Nov 02 21:30:23.563391 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:30:23.563363 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee0a0727_6bb6_4f61_82cd_80e292cbf912.slice/crio-059b5dbe24fe2e6361a5e7a97da50d0d25809cbcf07dd0ba513f09a9285ba5df WatchSource:0}: Error finding container 059b5dbe24fe2e6361a5e7a97da50d0d25809cbcf07dd0ba513f09a9285ba5df: Status 404 returned error can't find the container with id 059b5dbe24fe2e6361a5e7a97da50d0d25809cbcf07dd0ba513f09a9285ba5df Nov 02 21:30:23.566227 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:23.566197 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:30:23.566336 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:23.566275 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:30:23.566336 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:23.566320 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:30:23.771158 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:23.770551 4935 scope.go:122] "RemoveContainer" containerID="9f4d08cbb0526f27d33f6dedd76ba575ecbed9e8883d251b0f4eb4c3ed22db7f" Nov 02 21:30:24.041640 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:24.041590 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" event={"ID":"ee0a0727-6bb6-4f61-82cd-80e292cbf912","Type":"ContainerStarted","Data":"8be6618945007b5ec8a6ba35a9b10765e46bec8b259c909a3f6e5cfa75718c4a"} Nov 02 21:30:24.041834 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:24.041668 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" event={"ID":"ee0a0727-6bb6-4f61-82cd-80e292cbf912","Type":"ContainerStarted","Data":"059b5dbe24fe2e6361a5e7a97da50d0d25809cbcf07dd0ba513f09a9285ba5df"} Nov 02 21:30:24.096464 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:24.096423 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" event={"ID":"a290387e-9211-4298-a795-ac80d10f5ae9","Type":"ContainerStarted","Data":"de7017ec34a6a711a4cc568aaacbe96b0e200582722d52e4447890d55d30d749"} Nov 02 21:30:24.096725 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:24.096698 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" event={"ID":"a290387e-9211-4298-a795-ac80d10f5ae9","Type":"ContainerStarted","Data":"a45533ac57c91fa4996e1d335bcd0599b3c8ba8cd4cbc43496dc0483daa1a6c8"} Nov 02 21:30:24.119496 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:24.119450 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 02 21:30:24.119763 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:24.119720 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" podUID="ee0a0727-6bb6-4f61-82cd-80e292cbf912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 02 21:30:24.120282 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:24.120205 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:24.144019 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:24.140739 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:24.163378 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:24.163321 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" podStartSLOduration=2.163301764 podStartE2EDuration="2.163301764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:24.158966401 +0000 UTC m=+1316.598541126" watchObservedRunningTime="2026-11-02 21:30:24.163301764 +0000 UTC m=+1316.602876488" Nov 02 21:30:24.276169 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:24.273793 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" podStartSLOduration=2.273778405 podStartE2EDuration="2.273778405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:24.27163697 +0000 UTC m=+1316.711211693" watchObservedRunningTime="2026-11-02 21:30:24.273778405 +0000 UTC m=+1316.713353127" Nov 02 21:30:24.342800 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:24.342717 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:24.763218 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:24.762640 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:25.073464 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:25.073380 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:25.076268 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:25.076235 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 21:30:25.076480 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:25.076454 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" podUID="a290387e-9211-4298-a795-ac80d10f5ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 21:30:25.106111 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:25.104535 4935 generic.go:356] "Generic (PLEG): container finished" podID="42cfaf5c-edee-4b8c-b8e8-05bfda270667" containerID="4756659b3fe48b5420cd6860ea1c5ca7170b7abc67f1817592664202373fe591" exitCode=60 Nov 02 21:30:25.106111 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:25.105199 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-f7dhw" event={"ID":"42cfaf5c-edee-4b8c-b8e8-05bfda270667","Type":"ContainerDied","Data":"4756659b3fe48b5420cd6860ea1c5ca7170b7abc67f1817592664202373fe591"} Nov 02 21:30:25.106111 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:25.105257 4935 scope.go:122] "RemoveContainer" containerID="9f4d08cbb0526f27d33f6dedd76ba575ecbed9e8883d251b0f4eb4c3ed22db7f" Nov 02 21:30:25.106111 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:25.105674 4935 scope.go:122] "RemoveContainer" containerID="4756659b3fe48b5420cd6860ea1c5ca7170b7abc67f1817592664202373fe591" Nov 02 21:30:25.106111 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:30:25.105930 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894250-f7dhw_odh-ai-gateway-infra(42cfaf5c-edee-4b8c-b8e8-05bfda270667)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-f7dhw" podUID="42cfaf5c-edee-4b8c-b8e8-05bfda270667" Nov 02 21:30:25.117451 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:25.117429 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 02 21:30:25.117610 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:25.117592 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" podUID="ee0a0727-6bb6-4f61-82cd-80e292cbf912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 02 21:30:26.072156 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:26.071528 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 21:30:26.072156 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:26.071595 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" podUID="a290387e-9211-4298-a795-ac80d10f5ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 21:30:26.117365 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:26.117329 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 02 21:30:26.117559 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:26.117390 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" podUID="ee0a0727-6bb6-4f61-82cd-80e292cbf912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 02 21:30:26.228790 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:26.228756 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:26.256223 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:26.256182 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894250-f7dhw"] Nov 02 21:30:26.826157 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:26.821844 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-f7dhw" Nov 02 21:30:26.958750 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:26.958712 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42cfaf5c-edee-4b8c-b8e8-05bfda270667-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42cfaf5c-edee-4b8c-b8e8-05bfda270667-maas-api-service-ca\") pod \"42cfaf5c-edee-4b8c-b8e8-05bfda270667\" (UID: \"42cfaf5c-edee-4b8c-b8e8-05bfda270667\") " Nov 02 21:30:26.959025 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:26.959005 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42cfaf5c-edee-4b8c-b8e8-05bfda270667-kube-api-access-mr7n7\" (UniqueName: \"kubernetes.io/projected/42cfaf5c-edee-4b8c-b8e8-05bfda270667-kube-api-access-mr7n7\") pod \"42cfaf5c-edee-4b8c-b8e8-05bfda270667\" (UID: \"42cfaf5c-edee-4b8c-b8e8-05bfda270667\") " Nov 02 21:30:26.960469 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:26.960443 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42cfaf5c-edee-4b8c-b8e8-05bfda270667-maas-api-service-ca" pod "42cfaf5c-edee-4b8c-b8e8-05bfda270667" (UID: "42cfaf5c-edee-4b8c-b8e8-05bfda270667"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:30:26.991391 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:26.991351 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42cfaf5c-edee-4b8c-b8e8-05bfda270667-kube-api-access-mr7n7" pod "42cfaf5c-edee-4b8c-b8e8-05bfda270667" (UID: "42cfaf5c-edee-4b8c-b8e8-05bfda270667"). InnerVolumeSpecName "kube-api-access-mr7n7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:30:27.066476 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:27.061082 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42cfaf5c-edee-4b8c-b8e8-05bfda270667-maas-api-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:27.067281 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:27.067250 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr7n7\" (UniqueName: \"kubernetes.io/projected/42cfaf5c-edee-4b8c-b8e8-05bfda270667-kube-api-access-mr7n7\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:27.077363 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:27.077286 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 21:30:27.077893 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:27.077857 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" podUID="a290387e-9211-4298-a795-ac80d10f5ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 21:30:27.125250 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:27.125216 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 02 21:30:27.125519 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:27.125492 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" podUID="ee0a0727-6bb6-4f61-82cd-80e292cbf912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 02 21:30:27.290588 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:27.290541 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-f7dhw" event={"ID":"42cfaf5c-edee-4b8c-b8e8-05bfda270667","Type":"ContainerDied","Data":"54def84541a94c61712c58615921028b6194151403f1b907cfa915d2d3798011"} Nov 02 21:30:27.290925 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:27.290898 4935 scope.go:122] "RemoveContainer" containerID="4756659b3fe48b5420cd6860ea1c5ca7170b7abc67f1817592664202373fe591" Nov 02 21:30:27.291170 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:27.291155 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-f7dhw" Nov 02 21:30:27.501959 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:27.501925 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894250-f7dhw"] Nov 02 21:30:27.506084 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:27.506051 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894250-f7dhw"] Nov 02 21:30:27.797157 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:27.796194 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42cfaf5c-edee-4b8c-b8e8-05bfda270667" path="/var/lib/kubelet/pods/42cfaf5c-edee-4b8c-b8e8-05bfda270667/volumes" Nov 02 21:30:28.078536 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:28.074047 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 21:30:28.078536 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:28.074102 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" podUID="a290387e-9211-4298-a795-ac80d10f5ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 21:30:28.078536 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:28.074567 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:28.117773 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:28.117735 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 02 21:30:28.117915 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:28.117788 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" podUID="ee0a0727-6bb6-4f61-82cd-80e292cbf912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 02 21:30:28.501627 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:30:28.501541 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:30:28.750155 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:28.750000 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:29.077155 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:29.075253 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 21:30:29.077155 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:29.075329 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" podUID="a290387e-9211-4298-a795-ac80d10f5ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 21:30:29.118058 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:29.118023 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 02 21:30:29.118437 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:29.118075 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" podUID="ee0a0727-6bb6-4f61-82cd-80e292cbf912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 02 21:30:30.071810 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:30.071773 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 21:30:30.072019 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:30.071828 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" podUID="a290387e-9211-4298-a795-ac80d10f5ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 21:30:30.117291 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:30.117250 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 02 21:30:30.117483 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:30.117304 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" podUID="ee0a0727-6bb6-4f61-82cd-80e292cbf912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 02 21:30:31.071181 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:31.071111 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 21:30:31.071690 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:31.071208 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" podUID="a290387e-9211-4298-a795-ac80d10f5ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 21:30:31.546214 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:31.542784 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:31.546214 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:31.544568 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:31.567397 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:31.567360 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:30:32.075874 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:32.075843 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:32.076366 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:32.076346 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:32.077100 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:32.077081 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:30:33.769302 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:33.769266 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:33.978494 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:33.978459 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:34.321412 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:34.321378 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:39.028400 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:39.028303 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:40.068335 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.068298 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r"] Nov 02 21:30:40.069084 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.069066 4935 cpu_manager.go:414] "containerMap: removing container" podUID="42cfaf5c-edee-4b8c-b8e8-05bfda270667" containerName="cleanup" Nov 02 21:30:40.069154 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.069088 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42cfaf5c-edee-4b8c-b8e8-05bfda270667" containerName="cleanup" Nov 02 21:30:40.069200 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.069156 4935 cpu_manager.go:414] "containerMap: removing container" podUID="42cfaf5c-edee-4b8c-b8e8-05bfda270667" containerName="cleanup" Nov 02 21:30:40.069200 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.069165 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42cfaf5c-edee-4b8c-b8e8-05bfda270667" containerName="cleanup" Nov 02 21:30:40.069200 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.069177 4935 cpu_manager.go:414] "containerMap: removing container" podUID="42cfaf5c-edee-4b8c-b8e8-05bfda270667" containerName="cleanup" Nov 02 21:30:40.069200 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.069184 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42cfaf5c-edee-4b8c-b8e8-05bfda270667" containerName="cleanup" Nov 02 21:30:40.069331 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.069324 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="42cfaf5c-edee-4b8c-b8e8-05bfda270667" containerName="cleanup" Nov 02 21:30:40.069365 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.069340 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="42cfaf5c-edee-4b8c-b8e8-05bfda270667" containerName="cleanup" Nov 02 21:30:40.078125 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.078104 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.127055 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.127019 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-0c9014-openshift-default-dockercfg-7ll9g\"" Nov 02 21:30:40.140736 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.140708 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.140878 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.140749 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22263a6-d470-4d3f-962c-955cd7b003c9-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.140935 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.140911 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a22263a6-d470-4d3f-962c-955cd7b003c9-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.140999 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.140983 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22263a6-d470-4d3f-962c-955cd7b003c9-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.141155 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.141117 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.141205 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.141190 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.141264 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.141244 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.141344 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.141321 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.141445 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.141430 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfc5s\" (UniqueName: \"kubernetes.io/projected/a22263a6-d470-4d3f-962c-955cd7b003c9-kube-api-access-bfc5s\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.179599 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.179567 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r"] Nov 02 21:30:40.243170 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.243122 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfc5s\" (UniqueName: \"kubernetes.io/projected/a22263a6-d470-4d3f-962c-955cd7b003c9-kube-api-access-bfc5s\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.243345 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.243196 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.243345 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.243238 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22263a6-d470-4d3f-962c-955cd7b003c9-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.243345 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.243303 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a22263a6-d470-4d3f-962c-955cd7b003c9-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.243476 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.243359 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22263a6-d470-4d3f-962c-955cd7b003c9-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.243476 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.243416 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.243476 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.243465 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.243636 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.243509 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.243636 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.243543 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.244227 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.243927 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.244227 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.243948 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.244227 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.243971 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.244227 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.244173 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.244522 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.244388 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22263a6-d470-4d3f-962c-955cd7b003c9-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.245871 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.245847 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.246407 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.246388 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22263a6-d470-4d3f-962c-955cd7b003c9-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.297827 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.297800 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a22263a6-d470-4d3f-962c-955cd7b003c9-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.298030 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.298008 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfc5s\" (UniqueName: \"kubernetes.io/projected/a22263a6-d470-4d3f-962c-955cd7b003c9-kube-api-access-bfc5s\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.392379 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.392265 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:40.825493 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.825459 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r"] Nov 02 21:30:40.827506 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:30:40.827474 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda22263a6_d470_4d3f_962c_955cd7b003c9.slice/crio-c5fd95a6edbdf59cf6cfdbcc5c4f5d8e4fc87a59f579d53d12edf3276b388363 WatchSource:0}: Error finding container c5fd95a6edbdf59cf6cfdbcc5c4f5d8e4fc87a59f579d53d12edf3276b388363: Status 404 returned error can't find the container with id c5fd95a6edbdf59cf6cfdbcc5c4f5d8e4fc87a59f579d53d12edf3276b388363 Nov 02 21:30:40.830048 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.830015 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:30:40.830168 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.830097 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:30:40.830245 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:40.830176 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:30:41.408419 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:41.408386 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" event={"ID":"a22263a6-d470-4d3f-962c-955cd7b003c9","Type":"ContainerStarted","Data":"534a3f95412a8aa996372205742881b9c0ce212c31ac52b938d40b88224fe39d"} Nov 02 21:30:41.408870 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:41.408436 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" event={"ID":"a22263a6-d470-4d3f-962c-955cd7b003c9","Type":"ContainerStarted","Data":"c5fd95a6edbdf59cf6cfdbcc5c4f5d8e4fc87a59f579d53d12edf3276b388363"} Nov 02 21:30:41.597893 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:41.597827 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" podStartSLOduration=1.597812899 podStartE2EDuration="1.597812899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:41.563932861 +0000 UTC m=+1334.003507584" watchObservedRunningTime="2026-11-02 21:30:41.597812899 +0000 UTC m=+1334.037387620" Nov 02 21:30:42.392710 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:42.392669 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:42.394050 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:42.394022 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:30:42.394242 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:42.394086 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" podUID="a22263a6-d470-4d3f-962c-955cd7b003c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 02 21:30:43.393021 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:43.392982 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:30:43.393469 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:43.393054 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" podUID="a22263a6-d470-4d3f-962c-955cd7b003c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 02 21:30:44.204664 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:44.204591 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:44.392890 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:44.392853 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:30:44.393071 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:44.392912 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" podUID="a22263a6-d470-4d3f-962c-955cd7b003c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 02 21:30:45.498497 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:45.498463 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:30:45.499763 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:45.499730 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" podUID="a22263a6-d470-4d3f-962c-955cd7b003c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 02 21:30:46.298858 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:46.298742 4935 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" podUID="3920de75-9c3e-4826-8de5-22fcc6d50b0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:30:46.356756 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:46.356710 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" podUID="3920de75-9c3e-4826-8de5-22fcc6d50b0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:30:46.399318 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:46.399279 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:30:46.399968 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:46.399932 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" podUID="a22263a6-d470-4d3f-962c-955cd7b003c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 02 21:30:46.491356 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:46.491285 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" podUID="aab5db02-4890-45e1-bcf8-1c43c335a393" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 02 21:30:46.550243 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:46.550121 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" podUID="095af893-e150-43b9-b479-263a270d05c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:30:46.698392 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:46.698331 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" podUID="1d6a5790-dd1f-4721-bbde-3aa6b14999b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:30:46.699696 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:46.699557 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" podUID="7ba3a2e4-a47f-425a-8c65-5813ef0939c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:30:46.834328 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:46.834221 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" podUID="7cb3ac21-f737-4fc5-927e-4a8388582812" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 02 21:30:46.892478 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:46.892434 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" podUID="39b34814-d9a5-4505-9906-cd18e55eed4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 02 21:30:47.221334 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:47.221222 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" podUID="bcf6e677-5fc9-447b-821c-f8ba0ecd65f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:30:47.359342 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:47.359278 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" podUID="3920de75-9c3e-4826-8de5-22fcc6d50b0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 02 21:30:47.360479 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:47.360457 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" Nov 02 21:30:47.396562 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:47.396513 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:30:47.396837 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:47.396793 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" podUID="a22263a6-d470-4d3f-962c-955cd7b003c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 02 21:30:47.492781 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:47.491811 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" podUID="aab5db02-4890-45e1-bcf8-1c43c335a393" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:30:47.492781 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:47.492245 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" Nov 02 21:30:47.635276 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:47.576925 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" podUID="095af893-e150-43b9-b479-263a270d05c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:30:47.965160 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:47.826256 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" podUID="7ba3a2e4-a47f-425a-8c65-5813ef0939c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:30:47.976965 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:47.976902 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" podUID="1d6a5790-dd1f-4721-bbde-3aa6b14999b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 02 21:30:48.030532 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:48.030483 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" podUID="7cb3ac21-f737-4fc5-927e-4a8388582812" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 02 21:30:48.367488 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:48.360308 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" podUID="3920de75-9c3e-4826-8de5-22fcc6d50b0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:30:48.420292 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:48.420257 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" Nov 02 21:30:48.420602 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:48.420581 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" Nov 02 21:30:48.420760 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:48.420747 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" Nov 02 21:30:48.420893 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:48.420880 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" Nov 02 21:30:48.421031 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:48.421018 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" Nov 02 21:30:48.435511 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:48.435161 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:30:48.435737 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:48.435708 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" podUID="a22263a6-d470-4d3f-962c-955cd7b003c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 02 21:30:48.453675 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:48.453633 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" Nov 02 21:30:48.498520 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:48.498472 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" podUID="aab5db02-4890-45e1-bcf8-1c43c335a393" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:30:48.662853 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:48.662767 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" Nov 02 21:30:48.915242 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:48.907377 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" podUID="095af893-e150-43b9-b479-263a270d05c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:30:49.008116 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:48.982442 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" podUID="1d6a5790-dd1f-4721-bbde-3aa6b14999b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:30:49.044157 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:49.043847 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" podUID="39b34814-d9a5-4505-9906-cd18e55eed4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:30:49.055392 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:49.055362 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" Nov 02 21:30:49.393324 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:49.393282 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:30:49.393579 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:49.393548 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" podUID="a22263a6-d470-4d3f-962c-955cd7b003c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 02 21:30:49.542448 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:49.542404 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" Nov 02 21:30:49.591056 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:49.591025 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" Nov 02 21:30:49.996418 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:49.995626 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:51.434291 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:51.434253 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:30:51.434868 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:51.434833 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" podUID="a22263a6-d470-4d3f-962c-955cd7b003c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:30:51.513235 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:51.513173 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:51.515091 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:51.515061 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:51.516261 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:51.516237 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:30:54.261438 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:54.261402 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:54.496419 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:54.496379 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l"] Nov 02 21:30:54.497025 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:54.497003 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="42cfaf5c-edee-4b8c-b8e8-05bfda270667" containerName="cleanup" Nov 02 21:30:54.500748 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:54.500724 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" Nov 02 21:30:54.596059 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:54.596024 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f51151\"/\"distinct-w3-f51151-kserve-self-signed-certs\"" Nov 02 21:30:54.599291 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:54.599258 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l"] Nov 02 21:30:54.616710 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:54.616680 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91602668-a9dd-48f5-87cb-7b7ce484db3b-tmp-dir\") pod \"distinct-w3-f51151-kserve-846c54f579-lfs2l\" (UID: \"91602668-a9dd-48f5-87cb-7b7ce484db3b\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" Nov 02 21:30:54.616875 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:54.616751 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91602668-a9dd-48f5-87cb-7b7ce484db3b-home\") pod \"distinct-w3-f51151-kserve-846c54f579-lfs2l\" (UID: \"91602668-a9dd-48f5-87cb-7b7ce484db3b\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" Nov 02 21:30:54.616875 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:54.616828 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91602668-a9dd-48f5-87cb-7b7ce484db3b-tls-certs\") pod \"distinct-w3-f51151-kserve-846c54f579-lfs2l\" (UID: \"91602668-a9dd-48f5-87cb-7b7ce484db3b\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" Nov 02 21:30:54.616875 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:54.616865 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91602668-a9dd-48f5-87cb-7b7ce484db3b-dshm\") pod \"distinct-w3-f51151-kserve-846c54f579-lfs2l\" (UID: \"91602668-a9dd-48f5-87cb-7b7ce484db3b\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" Nov 02 21:30:54.617095 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:54.617013 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91602668-a9dd-48f5-87cb-7b7ce484db3b-model-cache\") pod \"distinct-w3-f51151-kserve-846c54f579-lfs2l\" (UID: \"91602668-a9dd-48f5-87cb-7b7ce484db3b\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" Nov 02 21:30:54.617176 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:54.617104 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds64t\" (UniqueName: \"kubernetes.io/projected/91602668-a9dd-48f5-87cb-7b7ce484db3b-kube-api-access-ds64t\") pod \"distinct-w3-f51151-kserve-846c54f579-lfs2l\" (UID: \"91602668-a9dd-48f5-87cb-7b7ce484db3b\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" Nov 02 21:30:54.663937 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:54.663896 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:54.718996 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:54.718955 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91602668-a9dd-48f5-87cb-7b7ce484db3b-home\") pod \"distinct-w3-f51151-kserve-846c54f579-lfs2l\" (UID: \"91602668-a9dd-48f5-87cb-7b7ce484db3b\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" Nov 02 21:30:54.719213 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:54.719017 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91602668-a9dd-48f5-87cb-7b7ce484db3b-tls-certs\") pod \"distinct-w3-f51151-kserve-846c54f579-lfs2l\" (UID: \"91602668-a9dd-48f5-87cb-7b7ce484db3b\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" Nov 02 21:30:54.719213 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:54.719062 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91602668-a9dd-48f5-87cb-7b7ce484db3b-dshm\") pod \"distinct-w3-f51151-kserve-846c54f579-lfs2l\" (UID: \"91602668-a9dd-48f5-87cb-7b7ce484db3b\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" Nov 02 21:30:54.719340 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:54.719221 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91602668-a9dd-48f5-87cb-7b7ce484db3b-model-cache\") pod \"distinct-w3-f51151-kserve-846c54f579-lfs2l\" (UID: \"91602668-a9dd-48f5-87cb-7b7ce484db3b\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" Nov 02 21:30:54.719340 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:54.719318 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds64t\" (UniqueName: \"kubernetes.io/projected/91602668-a9dd-48f5-87cb-7b7ce484db3b-kube-api-access-ds64t\") pod \"distinct-w3-f51151-kserve-846c54f579-lfs2l\" (UID: \"91602668-a9dd-48f5-87cb-7b7ce484db3b\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" Nov 02 21:30:54.719455 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:54.719359 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91602668-a9dd-48f5-87cb-7b7ce484db3b-tmp-dir\") pod \"distinct-w3-f51151-kserve-846c54f579-lfs2l\" (UID: \"91602668-a9dd-48f5-87cb-7b7ce484db3b\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" Nov 02 21:30:54.719906 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:54.719874 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91602668-a9dd-48f5-87cb-7b7ce484db3b-model-cache\") pod \"distinct-w3-f51151-kserve-846c54f579-lfs2l\" (UID: \"91602668-a9dd-48f5-87cb-7b7ce484db3b\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" Nov 02 21:30:54.720349 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:54.720324 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91602668-a9dd-48f5-87cb-7b7ce484db3b-home\") pod \"distinct-w3-f51151-kserve-846c54f579-lfs2l\" (UID: \"91602668-a9dd-48f5-87cb-7b7ce484db3b\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" Nov 02 21:30:54.720753 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:54.720730 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91602668-a9dd-48f5-87cb-7b7ce484db3b-tmp-dir\") pod \"distinct-w3-f51151-kserve-846c54f579-lfs2l\" (UID: \"91602668-a9dd-48f5-87cb-7b7ce484db3b\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" Nov 02 21:30:54.722986 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:54.722942 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91602668-a9dd-48f5-87cb-7b7ce484db3b-dshm\") pod \"distinct-w3-f51151-kserve-846c54f579-lfs2l\" (UID: \"91602668-a9dd-48f5-87cb-7b7ce484db3b\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" Nov 02 21:30:54.729770 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:54.729721 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91602668-a9dd-48f5-87cb-7b7ce484db3b-tls-certs\") pod \"distinct-w3-f51151-kserve-846c54f579-lfs2l\" (UID: \"91602668-a9dd-48f5-87cb-7b7ce484db3b\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" Nov 02 21:30:54.773184 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:54.773145 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds64t\" (UniqueName: \"kubernetes.io/projected/91602668-a9dd-48f5-87cb-7b7ce484db3b-kube-api-access-ds64t\") pod \"distinct-w3-f51151-kserve-846c54f579-lfs2l\" (UID: \"91602668-a9dd-48f5-87cb-7b7ce484db3b\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" Nov 02 21:30:54.815586 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:54.815546 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" Nov 02 21:30:54.908539 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:54.907694 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:55.122593 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:55.122534 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:55.329973 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:55.329934 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:55.361867 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:55.361828 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l"] Nov 02 21:30:55.363372 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:30:55.363342 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91602668_a9dd_48f5_87cb_7b7ce484db3b.slice/crio-20e6807d3127049e976fd76f3f9516c43a3b22b3e5701e8819cda5f91e0e7aca WatchSource:0}: Error finding container 20e6807d3127049e976fd76f3f9516c43a3b22b3e5701e8819cda5f91e0e7aca: Status 404 returned error can't find the container with id 20e6807d3127049e976fd76f3f9516c43a3b22b3e5701e8819cda5f91e0e7aca Nov 02 21:30:56.147395 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:56.147356 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" event={"ID":"91602668-a9dd-48f5-87cb-7b7ce484db3b","Type":"ContainerStarted","Data":"a488b0828abbb73269037cc2aec0ea7690e9339e6a1610f345e7371d15afcd6f"} Nov 02 21:30:56.147568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:56.147405 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" event={"ID":"91602668-a9dd-48f5-87cb-7b7ce484db3b","Type":"ContainerStarted","Data":"20e6807d3127049e976fd76f3f9516c43a3b22b3e5701e8819cda5f91e0e7aca"} Nov 02 21:30:56.253566 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:56.253514 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" podStartSLOduration=2.253498259 podStartE2EDuration="2.253498259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:56.251599127 +0000 UTC m=+1348.691173849" watchObservedRunningTime="2026-11-02 21:30:56.253498259 +0000 UTC m=+1348.693072980" Nov 02 21:30:56.816506 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:56.816473 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" Nov 02 21:30:58.470732 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:30:58.470692 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:30:59.053072 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:59.053012 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:30:59.271637 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:30:59.271598 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:00.454959 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:00.454922 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:00.659600 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:00.659564 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:02.225174 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:02.225117 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:02.462571 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:02.462495 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:04.119862 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:04.119820 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:04.803500 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:04.803458 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:04.817757 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:04.816808 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" Nov 02 21:31:04.838157 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:04.836103 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" Nov 02 21:31:05.218717 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:05.218619 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" Nov 02 21:31:05.220045 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:05.219984 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:05.859911 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:05.858826 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:08.820101 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:08.820054 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:09.073227 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:09.070887 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:10.360156 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:10.359500 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:10.632948 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:10.632856 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:11.460181 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:11.459305 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:11.956356 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:11.956316 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:12.155655 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:12.155621 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:16.821612 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:16.821576 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:18.930339 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:18.930302 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:22.407121 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:22.407069 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:25.129774 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:25.129734 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:27.521969 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:27.521926 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:28.470524 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:31:28.470480 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:31:32.526265 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:32.526218 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:37.275549 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:37.275495 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8"] Nov 02 21:31:37.281765 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:37.281740 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8" Nov 02 21:31:37.331429 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:37.331397 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8"] Nov 02 21:31:37.355543 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:37.355509 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d58d18-serving-cert\"" Nov 02 21:31:37.419701 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:37.419666 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a33cd12-7bf8-4b6a-af93-201a73a69240-maas-api-tls\") pod \"maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8\" (UID: \"6a33cd12-7bf8-4b6a-af93-201a73a69240\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8" Nov 02 21:31:37.419882 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:37.419740 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkqqx\" (UniqueName: \"kubernetes.io/projected/6a33cd12-7bf8-4b6a-af93-201a73a69240-kube-api-access-vkqqx\") pod \"maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8\" (UID: \"6a33cd12-7bf8-4b6a-af93-201a73a69240\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8" Nov 02 21:31:37.419882 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:37.419850 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a33cd12-7bf8-4b6a-af93-201a73a69240-metrics-certs\") pod \"maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8\" (UID: \"6a33cd12-7bf8-4b6a-af93-201a73a69240\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8" Nov 02 21:31:37.419995 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:37.419880 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a33cd12-7bf8-4b6a-af93-201a73a69240-openshift-service-ca\") pod \"maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8\" (UID: \"6a33cd12-7bf8-4b6a-af93-201a73a69240\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8" Nov 02 21:31:37.520733 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:37.520699 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a33cd12-7bf8-4b6a-af93-201a73a69240-maas-api-tls\") pod \"maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8\" (UID: \"6a33cd12-7bf8-4b6a-af93-201a73a69240\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8" Nov 02 21:31:37.520933 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:37.520770 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkqqx\" (UniqueName: \"kubernetes.io/projected/6a33cd12-7bf8-4b6a-af93-201a73a69240-kube-api-access-vkqqx\") pod \"maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8\" (UID: \"6a33cd12-7bf8-4b6a-af93-201a73a69240\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8" Nov 02 21:31:37.520933 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:37.520914 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a33cd12-7bf8-4b6a-af93-201a73a69240-metrics-certs\") pod \"maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8\" (UID: \"6a33cd12-7bf8-4b6a-af93-201a73a69240\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8" Nov 02 21:31:37.520933 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:37.520933 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a33cd12-7bf8-4b6a-af93-201a73a69240-openshift-service-ca\") pod \"maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8\" (UID: \"6a33cd12-7bf8-4b6a-af93-201a73a69240\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8" Nov 02 21:31:37.521599 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:37.521576 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a33cd12-7bf8-4b6a-af93-201a73a69240-openshift-service-ca\") pod \"maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8\" (UID: \"6a33cd12-7bf8-4b6a-af93-201a73a69240\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8" Nov 02 21:31:37.523333 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:37.523307 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a33cd12-7bf8-4b6a-af93-201a73a69240-metrics-certs\") pod \"maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8\" (UID: \"6a33cd12-7bf8-4b6a-af93-201a73a69240\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8" Nov 02 21:31:37.523525 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:37.523502 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a33cd12-7bf8-4b6a-af93-201a73a69240-maas-api-tls\") pod \"maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8\" (UID: \"6a33cd12-7bf8-4b6a-af93-201a73a69240\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8" Nov 02 21:31:37.550662 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:37.550589 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:37.573710 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:37.573681 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkqqx\" (UniqueName: \"kubernetes.io/projected/6a33cd12-7bf8-4b6a-af93-201a73a69240-kube-api-access-vkqqx\") pod \"maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8\" (UID: \"6a33cd12-7bf8-4b6a-af93-201a73a69240\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8" Nov 02 21:31:37.595613 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:37.595587 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8" Nov 02 21:31:37.781939 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:37.781876 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:37.969089 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:37.969047 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8"] Nov 02 21:31:37.971360 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:31:37.971332 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a33cd12_7bf8_4b6a_af93_201a73a69240.slice/crio-3080db0bececd131ad8113ab42a2e360d5d6e5b945f29d36ea4ae5ad2dbe1945 WatchSource:0}: Error finding container 3080db0bececd131ad8113ab42a2e360d5d6e5b945f29d36ea4ae5ad2dbe1945: Status 404 returned error can't find the container with id 3080db0bececd131ad8113ab42a2e360d5d6e5b945f29d36ea4ae5ad2dbe1945 Nov 02 21:31:38.491850 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:38.491811 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8" event={"ID":"6a33cd12-7bf8-4b6a-af93-201a73a69240","Type":"ContainerStarted","Data":"3080db0bececd131ad8113ab42a2e360d5d6e5b945f29d36ea4ae5ad2dbe1945"} Nov 02 21:31:39.504286 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:39.504244 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8" event={"ID":"6a33cd12-7bf8-4b6a-af93-201a73a69240","Type":"ContainerStarted","Data":"0a43f4d59b80875877f06d4618048067c99dbc7e8e5a071a8e472f19f436af14"} Nov 02 21:31:39.504821 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:39.504398 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8" Nov 02 21:31:39.615935 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:39.615877 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8" podStartSLOduration=1.68303366 podStartE2EDuration="2.615862214s" totalImagesPullingTime="932.828554ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:31:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:31:39.613829825 +0000 UTC m=+1392.053404548" watchObservedRunningTime="2026-11-02 21:31:39.615862214 +0000 UTC m=+1392.055436936" Nov 02 21:31:42.038319 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.038275 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc"] Nov 02 21:31:42.042346 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.042328 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" Nov 02 21:31:42.088895 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.088860 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f58a753c-53b9-4570-8875-6123f00fed56-tls-certs\") pod \"e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc\" (UID: \"f58a753c-53b9-4570-8875-6123f00fed56\") " pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" Nov 02 21:31:42.089068 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.088932 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f58a753c-53b9-4570-8875-6123f00fed56-home\") pod \"e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc\" (UID: \"f58a753c-53b9-4570-8875-6123f00fed56\") " pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" Nov 02 21:31:42.089229 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.089199 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f58a753c-53b9-4570-8875-6123f00fed56-tmp-dir\") pod \"e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc\" (UID: \"f58a753c-53b9-4570-8875-6123f00fed56\") " pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" Nov 02 21:31:42.089321 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.089304 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f58a753c-53b9-4570-8875-6123f00fed56-dshm\") pod \"e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc\" (UID: \"f58a753c-53b9-4570-8875-6123f00fed56\") " pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" Nov 02 21:31:42.089367 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.089336 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f58a753c-53b9-4570-8875-6123f00fed56-model-cache\") pod \"e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc\" (UID: \"f58a753c-53b9-4570-8875-6123f00fed56\") " pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" Nov 02 21:31:42.089571 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.089548 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nstfd\" (UniqueName: \"kubernetes.io/projected/f58a753c-53b9-4570-8875-6123f00fed56-kube-api-access-nstfd\") pod \"e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc\" (UID: \"f58a753c-53b9-4570-8875-6123f00fed56\") " pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" Nov 02 21:31:42.093598 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.093570 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc"] Nov 02 21:31:42.136195 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.136161 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-70efbbab\"/\"openshift-service-ca.crt\"" Nov 02 21:31:42.136364 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.136200 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-70efbbab\"/\"kube-root-ca.crt\"" Nov 02 21:31:42.137472 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.137457 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-70efbbab\"/\"default-dockercfg-7klts\"" Nov 02 21:31:42.137550 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.137523 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-70efbbab\"/\"e2e-lifecycle-model-70efbbab-kserve-self-signed-certs\"" Nov 02 21:31:42.196716 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.196681 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:42.197810 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.197777 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nstfd\" (UniqueName: \"kubernetes.io/projected/f58a753c-53b9-4570-8875-6123f00fed56-kube-api-access-nstfd\") pod \"e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc\" (UID: \"f58a753c-53b9-4570-8875-6123f00fed56\") " pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" Nov 02 21:31:42.197971 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.197859 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f58a753c-53b9-4570-8875-6123f00fed56-tls-certs\") pod \"e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc\" (UID: \"f58a753c-53b9-4570-8875-6123f00fed56\") " pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" Nov 02 21:31:42.197971 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.197937 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f58a753c-53b9-4570-8875-6123f00fed56-home\") pod \"e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc\" (UID: \"f58a753c-53b9-4570-8875-6123f00fed56\") " pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" Nov 02 21:31:42.198093 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.198076 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f58a753c-53b9-4570-8875-6123f00fed56-tmp-dir\") pod \"e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc\" (UID: \"f58a753c-53b9-4570-8875-6123f00fed56\") " pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" Nov 02 21:31:42.198190 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.198147 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f58a753c-53b9-4570-8875-6123f00fed56-dshm\") pod \"e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc\" (UID: \"f58a753c-53b9-4570-8875-6123f00fed56\") " pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" Nov 02 21:31:42.198190 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.198174 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f58a753c-53b9-4570-8875-6123f00fed56-model-cache\") pod \"e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc\" (UID: \"f58a753c-53b9-4570-8875-6123f00fed56\") " pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" Nov 02 21:31:42.198737 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.198710 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f58a753c-53b9-4570-8875-6123f00fed56-model-cache\") pod \"e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc\" (UID: \"f58a753c-53b9-4570-8875-6123f00fed56\") " pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" Nov 02 21:31:42.200175 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.200151 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f58a753c-53b9-4570-8875-6123f00fed56-tmp-dir\") pod \"e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc\" (UID: \"f58a753c-53b9-4570-8875-6123f00fed56\") " pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" Nov 02 21:31:42.202227 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.200439 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f58a753c-53b9-4570-8875-6123f00fed56-home\") pod \"e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc\" (UID: \"f58a753c-53b9-4570-8875-6123f00fed56\") " pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" Nov 02 21:31:42.203735 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.203708 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f58a753c-53b9-4570-8875-6123f00fed56-dshm\") pod \"e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc\" (UID: \"f58a753c-53b9-4570-8875-6123f00fed56\") " pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" Nov 02 21:31:42.205678 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.205647 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f58a753c-53b9-4570-8875-6123f00fed56-tls-certs\") pod \"e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc\" (UID: \"f58a753c-53b9-4570-8875-6123f00fed56\") " pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" Nov 02 21:31:42.252533 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.252496 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nstfd\" (UniqueName: \"kubernetes.io/projected/f58a753c-53b9-4570-8875-6123f00fed56-kube-api-access-nstfd\") pod \"e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc\" (UID: \"f58a753c-53b9-4570-8875-6123f00fed56\") " pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" Nov 02 21:31:42.355990 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.355902 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" Nov 02 21:31:42.745775 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.745738 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc"] Nov 02 21:31:42.747405 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:31:42.747365 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf58a753c_53b9_4570_8875_6123f00fed56.slice/crio-d550a5c9e7da9b37ac176154e91fd187a41b007fe67cfb91498b310fcabe7503 WatchSource:0}: Error finding container d550a5c9e7da9b37ac176154e91fd187a41b007fe67cfb91498b310fcabe7503: Status 404 returned error can't find the container with id d550a5c9e7da9b37ac176154e91fd187a41b007fe67cfb91498b310fcabe7503 Nov 02 21:31:42.838493 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:42.838424 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:43.539833 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:43.539798 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" event={"ID":"f58a753c-53b9-4570-8875-6123f00fed56","Type":"ContainerStarted","Data":"e1a1d1290e970cd9dcd8e73aab18beabb3d086f3c391f21b6553e68bc5cb33f8"} Nov 02 21:31:43.539833 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:43.539839 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" event={"ID":"f58a753c-53b9-4570-8875-6123f00fed56","Type":"ContainerStarted","Data":"d550a5c9e7da9b37ac176154e91fd187a41b007fe67cfb91498b310fcabe7503"} Nov 02 21:31:43.649358 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:43.649298 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" podStartSLOduration=2.6492829369999997 podStartE2EDuration="2.649282937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:31:43.646212693 +0000 UTC m=+1396.085787419" watchObservedRunningTime="2026-11-02 21:31:43.649282937 +0000 UTC m=+1396.088857659" Nov 02 21:31:44.356440 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:44.356401 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" Nov 02 21:31:45.516182 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:45.516123 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8" Nov 02 21:31:47.916818 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:47.916785 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:52.356638 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:52.356602 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" Nov 02 21:31:52.369426 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:52.369399 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" Nov 02 21:31:52.581863 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:52.581828 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5"] Nov 02 21:31:52.588229 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:52.588180 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5" Nov 02 21:31:52.615510 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:52.615446 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" Nov 02 21:31:52.637678 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:52.637647 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-03ece2-serving-cert\"" Nov 02 21:31:52.638881 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:52.638856 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5"] Nov 02 21:31:52.748386 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:52.748353 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b66e9095-274e-449b-b895-8c7b973b2f47-metrics-certs\") pod \"maas-api-e2e-api-a-03ece2-56666dd645-wg9m5\" (UID: \"b66e9095-274e-449b-b895-8c7b973b2f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5" Nov 02 21:31:52.748577 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:52.748556 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66e9095-274e-449b-b895-8c7b973b2f47-openshift-service-ca\") pod \"maas-api-e2e-api-a-03ece2-56666dd645-wg9m5\" (UID: \"b66e9095-274e-449b-b895-8c7b973b2f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5" Nov 02 21:31:52.748788 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:52.748770 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b66e9095-274e-449b-b895-8c7b973b2f47-maas-api-tls\") pod \"maas-api-e2e-api-a-03ece2-56666dd645-wg9m5\" (UID: \"b66e9095-274e-449b-b895-8c7b973b2f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5" Nov 02 21:31:52.748825 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:52.748796 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p29k8\" (UniqueName: \"kubernetes.io/projected/b66e9095-274e-449b-b895-8c7b973b2f47-kube-api-access-p29k8\") pod \"maas-api-e2e-api-a-03ece2-56666dd645-wg9m5\" (UID: \"b66e9095-274e-449b-b895-8c7b973b2f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5" Nov 02 21:31:52.850205 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:52.850171 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b66e9095-274e-449b-b895-8c7b973b2f47-maas-api-tls\") pod \"maas-api-e2e-api-a-03ece2-56666dd645-wg9m5\" (UID: \"b66e9095-274e-449b-b895-8c7b973b2f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5" Nov 02 21:31:52.850375 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:52.850216 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p29k8\" (UniqueName: \"kubernetes.io/projected/b66e9095-274e-449b-b895-8c7b973b2f47-kube-api-access-p29k8\") pod \"maas-api-e2e-api-a-03ece2-56666dd645-wg9m5\" (UID: \"b66e9095-274e-449b-b895-8c7b973b2f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5" Nov 02 21:31:52.850375 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:52.850330 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b66e9095-274e-449b-b895-8c7b973b2f47-metrics-certs\") pod \"maas-api-e2e-api-a-03ece2-56666dd645-wg9m5\" (UID: \"b66e9095-274e-449b-b895-8c7b973b2f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5" Nov 02 21:31:52.850484 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:52.850427 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66e9095-274e-449b-b895-8c7b973b2f47-openshift-service-ca\") pod \"maas-api-e2e-api-a-03ece2-56666dd645-wg9m5\" (UID: \"b66e9095-274e-449b-b895-8c7b973b2f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5" Nov 02 21:31:52.851116 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:52.851088 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66e9095-274e-449b-b895-8c7b973b2f47-openshift-service-ca\") pod \"maas-api-e2e-api-a-03ece2-56666dd645-wg9m5\" (UID: \"b66e9095-274e-449b-b895-8c7b973b2f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5" Nov 02 21:31:52.853092 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:52.853061 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b66e9095-274e-449b-b895-8c7b973b2f47-metrics-certs\") pod \"maas-api-e2e-api-a-03ece2-56666dd645-wg9m5\" (UID: \"b66e9095-274e-449b-b895-8c7b973b2f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5" Nov 02 21:31:52.853278 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:52.853246 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:52.853278 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:52.853266 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b66e9095-274e-449b-b895-8c7b973b2f47-maas-api-tls\") pod \"maas-api-e2e-api-a-03ece2-56666dd645-wg9m5\" (UID: \"b66e9095-274e-449b-b895-8c7b973b2f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5" Nov 02 21:31:52.903450 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:52.903379 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p29k8\" (UniqueName: \"kubernetes.io/projected/b66e9095-274e-449b-b895-8c7b973b2f47-kube-api-access-p29k8\") pod \"maas-api-e2e-api-a-03ece2-56666dd645-wg9m5\" (UID: \"b66e9095-274e-449b-b895-8c7b973b2f47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5" Nov 02 21:31:52.911703 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:52.911679 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5" Nov 02 21:31:53.051974 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:53.051925 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:53.255373 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:53.255295 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:53.341399 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:53.341373 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5"] Nov 02 21:31:53.343440 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:31:53.343407 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb66e9095_274e_449b_b895_8c7b973b2f47.slice/crio-229c194f8bb9c5632b7b82ff05603d33c43f1f5459077908ded2470741166e90 WatchSource:0}: Error finding container 229c194f8bb9c5632b7b82ff05603d33c43f1f5459077908ded2470741166e90: Status 404 returned error can't find the container with id 229c194f8bb9c5632b7b82ff05603d33c43f1f5459077908ded2470741166e90 Nov 02 21:31:53.610735 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:53.610697 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5" event={"ID":"b66e9095-274e-449b-b895-8c7b973b2f47","Type":"ContainerStarted","Data":"229c194f8bb9c5632b7b82ff05603d33c43f1f5459077908ded2470741166e90"} Nov 02 21:31:54.620778 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:54.620685 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5" event={"ID":"b66e9095-274e-449b-b895-8c7b973b2f47","Type":"ContainerStarted","Data":"018e154ca3dd432fc10c69d93d7321dcc2c955c79e293646a0000332712e6e9e"} Nov 02 21:31:54.621161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:54.620830 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5" Nov 02 21:31:58.121237 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:31:58.121180 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:31:58.474215 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:31:58.474117 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:32:00.634027 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:00.633996 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5" Nov 02 21:32:00.747432 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:00.747374 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5" podStartSLOduration=7.829117308 podStartE2EDuration="8.747360901s" totalImagesPullingTime="918.243593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:31:54.730351181 +0000 UTC m=+1407.169925904" watchObservedRunningTime="2026-11-02 21:32:00.747360901 +0000 UTC m=+1413.186935622" Nov 02 21:32:03.246645 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:03.246592 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:03.914580 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:03.914539 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:08.284382 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:08.284350 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:08.489752 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:08.489705 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:10.331471 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.331431 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8"] Nov 02 21:32:10.335664 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.335641 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:10.382443 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.382408 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istio-envoy\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:10.382606 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.382471 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-workload-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:10.382652 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.382620 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cq5b\" (UniqueName: \"kubernetes.io/projected/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-kube-api-access-4cq5b\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:10.382688 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.382676 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istio-token\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:10.382788 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.382767 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-workload-certs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:10.382823 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.382804 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istio-podinfo\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:10.382870 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.382858 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istiod-ca-cert\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:10.382925 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.382911 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istio-data\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:10.382960 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.382941 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-credential-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:10.384010 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.383995 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d58d18-openshift-default-dockercfg-6dd47\"" Nov 02 21:32:10.484671 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.484634 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istio-data\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:10.484853 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.484681 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-credential-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:10.484853 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.484713 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istio-envoy\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:10.484853 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.484764 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-workload-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:10.484986 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.484855 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cq5b\" (UniqueName: \"kubernetes.io/projected/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-kube-api-access-4cq5b\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:10.484986 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.484911 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istio-token\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:10.484986 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.484969 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-workload-certs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:10.485168 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.485003 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istio-podinfo\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:10.485168 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.485057 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istiod-ca-cert\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:10.485284 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.485215 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istio-data\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:10.485336 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.485315 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-workload-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:10.485560 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.485538 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-workload-certs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:10.485790 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.485759 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-credential-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:10.485882 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.485847 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istiod-ca-cert\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:10.487610 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.487577 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istio-envoy\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:10.487774 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.487754 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istio-podinfo\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:10.523050 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.523018 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8"] Nov 02 21:32:10.539302 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.539269 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cq5b\" (UniqueName: \"kubernetes.io/projected/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-kube-api-access-4cq5b\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:10.539518 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.539492 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istio-token\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:10.650889 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:10.650805 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:11.072049 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:11.072011 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8"] Nov 02 21:32:11.111225 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:32:11.111196 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba481e7_6f26_4ae3_91cf_0f2acea6fc33.slice/crio-6c1a0d0496db9ceef0040dcb050832ad49fdadb3fa7d18ab7ed1cc06b5d79479 WatchSource:0}: Error finding container 6c1a0d0496db9ceef0040dcb050832ad49fdadb3fa7d18ab7ed1cc06b5d79479: Status 404 returned error can't find the container with id 6c1a0d0496db9ceef0040dcb050832ad49fdadb3fa7d18ab7ed1cc06b5d79479 Nov 02 21:32:11.113721 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:11.113688 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:32:11.113837 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:11.113762 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:32:11.113837 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:11.113795 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:32:11.756733 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:11.756693 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" event={"ID":"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33","Type":"ContainerStarted","Data":"e141428f1d65ff4b072fdffb25e7be307644fdeca98631b39e6193f5685ba17f"} Nov 02 21:32:11.757197 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:11.756758 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" event={"ID":"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33","Type":"ContainerStarted","Data":"6c1a0d0496db9ceef0040dcb050832ad49fdadb3fa7d18ab7ed1cc06b5d79479"} Nov 02 21:32:11.965947 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:11.965889 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" podStartSLOduration=1.9658747060000001 podStartE2EDuration="1.965874706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:32:11.916689942 +0000 UTC m=+1424.356264675" watchObservedRunningTime="2026-11-02 21:32:11.965874706 +0000 UTC m=+1424.405449427" Nov 02 21:32:12.167746 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:12.167713 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:12.651520 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:12.651485 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:12.655908 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:12.655419 4935 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 21:32:12.655908 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:12.655493 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" podUID="5ba481e7-6f26-4ae3-91cf-0f2acea6fc33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 21:32:13.391910 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:13.391865 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:13.651860 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:13.651761 4935 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 21:32:13.651860 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:13.651821 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" podUID="5ba481e7-6f26-4ae3-91cf-0f2acea6fc33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 21:32:14.651756 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:14.651713 4935 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 21:32:14.652214 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:14.651776 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" podUID="5ba481e7-6f26-4ae3-91cf-0f2acea6fc33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 21:32:15.651718 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:15.651668 4935 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 21:32:15.652064 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:15.652010 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" podUID="5ba481e7-6f26-4ae3-91cf-0f2acea6fc33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 21:32:16.651734 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:16.651691 4935 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 21:32:16.651923 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:16.651766 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" podUID="5ba481e7-6f26-4ae3-91cf-0f2acea6fc33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 21:32:17.656385 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:17.656350 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:17.656847 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:17.656806 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:17.657610 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:17.657589 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:32:18.434496 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:18.434428 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:21.778168 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:21.778121 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx"] Nov 02 21:32:21.782165 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:21.782144 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" Nov 02 21:32:21.855169 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:21.855115 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksmdg\" (UniqueName: \"kubernetes.io/projected/4534e64e-1398-44a8-acb0-22db48ebea02-kube-api-access-ksmdg\") pod \"distinct-2-w3-f51151-kserve-648bf5745b-xj7wx\" (UID: \"4534e64e-1398-44a8-acb0-22db48ebea02\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" Nov 02 21:32:21.855478 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:21.855431 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4534e64e-1398-44a8-acb0-22db48ebea02-tls-certs\") pod \"distinct-2-w3-f51151-kserve-648bf5745b-xj7wx\" (UID: \"4534e64e-1398-44a8-acb0-22db48ebea02\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" Nov 02 21:32:21.855595 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:21.855479 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4534e64e-1398-44a8-acb0-22db48ebea02-tmp-dir\") pod \"distinct-2-w3-f51151-kserve-648bf5745b-xj7wx\" (UID: \"4534e64e-1398-44a8-acb0-22db48ebea02\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" Nov 02 21:32:21.855675 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:21.855649 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4534e64e-1398-44a8-acb0-22db48ebea02-dshm\") pod \"distinct-2-w3-f51151-kserve-648bf5745b-xj7wx\" (UID: \"4534e64e-1398-44a8-acb0-22db48ebea02\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" Nov 02 21:32:21.855814 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:21.855785 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4534e64e-1398-44a8-acb0-22db48ebea02-home\") pod \"distinct-2-w3-f51151-kserve-648bf5745b-xj7wx\" (UID: \"4534e64e-1398-44a8-acb0-22db48ebea02\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" Nov 02 21:32:21.855897 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:21.855828 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4534e64e-1398-44a8-acb0-22db48ebea02-model-cache\") pod \"distinct-2-w3-f51151-kserve-648bf5745b-xj7wx\" (UID: \"4534e64e-1398-44a8-acb0-22db48ebea02\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" Nov 02 21:32:21.882065 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:21.882024 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f51151\"/\"distinct-2-w3-f51151-kserve-self-signed-certs\"" Nov 02 21:32:21.887886 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:21.887852 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx"] Nov 02 21:32:21.957097 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:21.957046 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksmdg\" (UniqueName: \"kubernetes.io/projected/4534e64e-1398-44a8-acb0-22db48ebea02-kube-api-access-ksmdg\") pod \"distinct-2-w3-f51151-kserve-648bf5745b-xj7wx\" (UID: \"4534e64e-1398-44a8-acb0-22db48ebea02\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" Nov 02 21:32:21.957341 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:21.957118 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4534e64e-1398-44a8-acb0-22db48ebea02-tls-certs\") pod \"distinct-2-w3-f51151-kserve-648bf5745b-xj7wx\" (UID: \"4534e64e-1398-44a8-acb0-22db48ebea02\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" Nov 02 21:32:21.957341 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:21.957192 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4534e64e-1398-44a8-acb0-22db48ebea02-tmp-dir\") pod \"distinct-2-w3-f51151-kserve-648bf5745b-xj7wx\" (UID: \"4534e64e-1398-44a8-acb0-22db48ebea02\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" Nov 02 21:32:21.957341 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:21.957278 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4534e64e-1398-44a8-acb0-22db48ebea02-dshm\") pod \"distinct-2-w3-f51151-kserve-648bf5745b-xj7wx\" (UID: \"4534e64e-1398-44a8-acb0-22db48ebea02\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" Nov 02 21:32:21.957512 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:21.957369 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4534e64e-1398-44a8-acb0-22db48ebea02-home\") pod \"distinct-2-w3-f51151-kserve-648bf5745b-xj7wx\" (UID: \"4534e64e-1398-44a8-acb0-22db48ebea02\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" Nov 02 21:32:21.957512 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:21.957395 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4534e64e-1398-44a8-acb0-22db48ebea02-model-cache\") pod \"distinct-2-w3-f51151-kserve-648bf5745b-xj7wx\" (UID: \"4534e64e-1398-44a8-acb0-22db48ebea02\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" Nov 02 21:32:21.968819 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:21.957865 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4534e64e-1398-44a8-acb0-22db48ebea02-model-cache\") pod \"distinct-2-w3-f51151-kserve-648bf5745b-xj7wx\" (UID: \"4534e64e-1398-44a8-acb0-22db48ebea02\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" Nov 02 21:32:21.968819 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:21.958350 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4534e64e-1398-44a8-acb0-22db48ebea02-home\") pod \"distinct-2-w3-f51151-kserve-648bf5745b-xj7wx\" (UID: \"4534e64e-1398-44a8-acb0-22db48ebea02\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" Nov 02 21:32:21.968819 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:21.958412 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4534e64e-1398-44a8-acb0-22db48ebea02-tmp-dir\") pod \"distinct-2-w3-f51151-kserve-648bf5745b-xj7wx\" (UID: \"4534e64e-1398-44a8-acb0-22db48ebea02\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" Nov 02 21:32:21.968819 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:21.965413 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4534e64e-1398-44a8-acb0-22db48ebea02-dshm\") pod \"distinct-2-w3-f51151-kserve-648bf5745b-xj7wx\" (UID: \"4534e64e-1398-44a8-acb0-22db48ebea02\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" Nov 02 21:32:21.974028 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:21.973995 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4534e64e-1398-44a8-acb0-22db48ebea02-tls-certs\") pod \"distinct-2-w3-f51151-kserve-648bf5745b-xj7wx\" (UID: \"4534e64e-1398-44a8-acb0-22db48ebea02\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" Nov 02 21:32:22.012481 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:22.012444 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:22.032751 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:22.032665 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksmdg\" (UniqueName: \"kubernetes.io/projected/4534e64e-1398-44a8-acb0-22db48ebea02-kube-api-access-ksmdg\") pod \"distinct-2-w3-f51151-kserve-648bf5745b-xj7wx\" (UID: \"4534e64e-1398-44a8-acb0-22db48ebea02\") " pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" Nov 02 21:32:22.095936 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:22.095888 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" Nov 02 21:32:22.513207 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:32:22.513174 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4534e64e_1398_44a8_acb0_22db48ebea02.slice/crio-5dd63669b94517ae7655ff4c8f45ff54a9f294fd6521c47b06fead03f769478d WatchSource:0}: Error finding container 5dd63669b94517ae7655ff4c8f45ff54a9f294fd6521c47b06fead03f769478d: Status 404 returned error can't find the container with id 5dd63669b94517ae7655ff4c8f45ff54a9f294fd6521c47b06fead03f769478d Nov 02 21:32:22.513326 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:22.513267 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx"] Nov 02 21:32:22.834788 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:22.834753 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" event={"ID":"4534e64e-1398-44a8-acb0-22db48ebea02","Type":"ContainerStarted","Data":"7cd7c96e3586a3fddbfeeee1c6cb10a14875ca6e47ad4625dd90e65329b64ed4"} Nov 02 21:32:22.835296 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:22.834797 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" event={"ID":"4534e64e-1398-44a8-acb0-22db48ebea02","Type":"ContainerStarted","Data":"5dd63669b94517ae7655ff4c8f45ff54a9f294fd6521c47b06fead03f769478d"} Nov 02 21:32:22.946195 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:22.946107 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" podStartSLOduration=1.946088247 podStartE2EDuration="1.946088247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:32:22.942971483 +0000 UTC m=+1435.382546207" watchObservedRunningTime="2026-11-02 21:32:22.946088247 +0000 UTC m=+1435.385662971" Nov 02 21:32:23.096187 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:23.096081 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" Nov 02 21:32:23.645993 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:23.645951 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:25.541429 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:25.541394 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:28.470362 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:32:28.470326 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:32:28.852976 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:28.852176 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:30.586300 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:30.586252 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:31.403741 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:31.403702 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:32.096226 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:32.096193 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" Nov 02 21:32:32.113994 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:32.113961 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" Nov 02 21:32:32.264146 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:32.264093 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:32.492176 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:32.492025 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:32.697858 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:32.697824 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:32.939657 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:32.939624 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" Nov 02 21:32:34.228230 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:34.228183 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:36.769029 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:36.768960 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:37.000286 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:37.000250 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:37.222714 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:37.222569 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:39.639969 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:39.639935 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:40.038303 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.038267 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:40.241739 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.241697 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:40.545172 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.545100 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p"] Nov 02 21:32:40.551457 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.551430 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:40.591740 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.591702 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs"] Nov 02 21:32:40.596410 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.596386 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:40.660364 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.660329 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p"] Nov 02 21:32:40.682335 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.682298 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs"] Nov 02 21:32:40.763444 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.763411 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-istio-envoy\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:40.763732 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.763717 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-credential-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:40.763871 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.763860 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-istio-data\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:40.763979 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.763969 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f896b6d0-edd2-4201-a17f-3671ef1517b7-istiod-ca-cert\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:40.764092 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.764082 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f896b6d0-edd2-4201-a17f-3671ef1517b7-istio-podinfo\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:40.764238 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.764217 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-workload-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:40.764320 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.764285 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-workload-certs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:40.764379 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.764362 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istio-envoy\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:40.764485 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.764473 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istiod-ca-cert\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:40.764608 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.764594 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istio-data\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:40.764692 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.764682 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istio-podinfo\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:40.764804 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.764794 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-workload-certs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:40.764902 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.764889 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dscvg\" (UniqueName: \"kubernetes.io/projected/f896b6d0-edd2-4201-a17f-3671ef1517b7-kube-api-access-dscvg\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:40.764973 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.764962 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-workload-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:40.765089 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.765078 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-credential-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:40.765189 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.765178 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f896b6d0-edd2-4201-a17f-3671ef1517b7-istio-token\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:40.765277 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.765268 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wczz9\" (UniqueName: \"kubernetes.io/projected/9f07e6bf-04ee-4b56-b10b-ee78424e956c-kube-api-access-wczz9\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:40.765393 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.765383 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istio-token\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:40.866680 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.866563 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istio-envoy\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:40.866680 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.866654 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istiod-ca-cert\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:40.866931 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.866705 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istio-data\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:40.866931 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.866735 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istio-podinfo\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:40.866931 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.866783 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-workload-certs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:40.866931 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.866843 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dscvg\" (UniqueName: \"kubernetes.io/projected/f896b6d0-edd2-4201-a17f-3671ef1517b7-kube-api-access-dscvg\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:40.866931 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.866867 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-workload-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:40.867219 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.866981 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-credential-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:40.867219 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.867021 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f896b6d0-edd2-4201-a17f-3671ef1517b7-istio-token\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:40.867219 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.867052 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wczz9\" (UniqueName: \"kubernetes.io/projected/9f07e6bf-04ee-4b56-b10b-ee78424e956c-kube-api-access-wczz9\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:40.867395 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.867355 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-workload-certs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:40.867453 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.867423 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istio-data\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:40.867771 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.867744 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-credential-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:40.868377 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.868319 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istio-token\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:40.868482 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.868411 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-istio-envoy\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:40.868482 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.868466 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-credential-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:40.868588 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.868479 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istiod-ca-cert\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:40.868588 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.868524 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-istio-data\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:40.868742 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.868714 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f896b6d0-edd2-4201-a17f-3671ef1517b7-istiod-ca-cert\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:40.868968 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.868945 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f896b6d0-edd2-4201-a17f-3671ef1517b7-istio-podinfo\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:40.869079 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.869053 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-credential-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:40.869282 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.869262 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-workload-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:40.869433 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.869416 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-workload-certs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:40.869759 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.869732 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f896b6d0-edd2-4201-a17f-3671ef1517b7-istiod-ca-cert\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:40.869990 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.869971 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-workload-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:40.870105 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.868735 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-workload-socket\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:40.870200 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.870170 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-workload-certs\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:40.870395 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.870371 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-istio-data\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:40.871056 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.871025 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istio-podinfo\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:40.871187 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.871095 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istio-envoy\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:40.871398 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.871369 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-istio-envoy\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:40.872563 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.872543 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f896b6d0-edd2-4201-a17f-3671ef1517b7-istio-podinfo\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:40.922056 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.922014 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dscvg\" (UniqueName: \"kubernetes.io/projected/f896b6d0-edd2-4201-a17f-3671ef1517b7-kube-api-access-dscvg\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:40.922261 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.922074 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f896b6d0-edd2-4201-a17f-3671ef1517b7-istio-token\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:40.922261 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.922251 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wczz9\" (UniqueName: \"kubernetes.io/projected/9f07e6bf-04ee-4b56-b10b-ee78424e956c-kube-api-access-wczz9\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:40.922614 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:40.922592 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istio-token\") pod \"e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:41.168683 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:41.168583 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:41.223633 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:41.223399 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:41.599608 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:41.599568 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p"] Nov 02 21:32:41.638368 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:32:41.638332 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf896b6d0_edd2_4201_a17f_3671ef1517b7.slice/crio-b2fa6d4984f713025af16c56b35a19419daed75dd6f41c90a26e6ae93c4c038d WatchSource:0}: Error finding container b2fa6d4984f713025af16c56b35a19419daed75dd6f41c90a26e6ae93c4c038d: Status 404 returned error can't find the container with id b2fa6d4984f713025af16c56b35a19419daed75dd6f41c90a26e6ae93c4c038d Nov 02 21:32:41.641067 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:41.641030 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:32:41.641225 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:41.641096 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:32:41.641225 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:41.641125 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:32:41.650435 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:41.650412 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs"] Nov 02 21:32:41.652311 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:32:41.652271 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f07e6bf_04ee_4b56_b10b_ee78424e956c.slice/crio-cdbb512678dd8e517d1fe6823fb314bd72949b7c89b66fcd6650a7f1d3051ead WatchSource:0}: Error finding container cdbb512678dd8e517d1fe6823fb314bd72949b7c89b66fcd6650a7f1d3051ead: Status 404 returned error can't find the container with id cdbb512678dd8e517d1fe6823fb314bd72949b7c89b66fcd6650a7f1d3051ead Nov 02 21:32:41.655884 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:41.655295 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:32:41.655884 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:41.655424 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:32:41.655884 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:41.655475 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:32:41.998735 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:41.998634 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" event={"ID":"f896b6d0-edd2-4201-a17f-3671ef1517b7","Type":"ContainerStarted","Data":"8530b83c064e4c4140f4592d4935aa62c300378971a82fbef3bb6b2d6c2b1f7c"} Nov 02 21:32:41.998735 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:41.998710 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" event={"ID":"f896b6d0-edd2-4201-a17f-3671ef1517b7","Type":"ContainerStarted","Data":"b2fa6d4984f713025af16c56b35a19419daed75dd6f41c90a26e6ae93c4c038d"} Nov 02 21:32:42.000958 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:42.000926 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" event={"ID":"9f07e6bf-04ee-4b56-b10b-ee78424e956c","Type":"ContainerStarted","Data":"fa6aaabac3bbb08c0b7f65282f3c1be4763bd6eab6035dd092b1010629bf2488"} Nov 02 21:32:42.001093 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:42.000986 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" event={"ID":"9f07e6bf-04ee-4b56-b10b-ee78424e956c","Type":"ContainerStarted","Data":"cdbb512678dd8e517d1fe6823fb314bd72949b7c89b66fcd6650a7f1d3051ead"} Nov 02 21:32:42.107612 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:42.107548 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" podStartSLOduration=2.107533394 podStartE2EDuration="2.107533394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:32:42.106616003 +0000 UTC m=+1454.546190724" watchObservedRunningTime="2026-11-02 21:32:42.107533394 +0000 UTC m=+1454.547108118" Nov 02 21:32:42.169217 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:42.169177 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:42.172161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:42.171856 4935 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 02 21:32:42.172161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:42.171914 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" podUID="f896b6d0-edd2-4201-a17f-3671ef1517b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 02 21:32:42.237002 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:42.236309 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:42.250209 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:42.246270 4935 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 21:32:42.250209 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:42.246331 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" podUID="9f07e6bf-04ee-4b56-b10b-ee78424e956c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 21:32:43.172154 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:43.171516 4935 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 02 21:32:43.172154 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:43.171579 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" podUID="f896b6d0-edd2-4201-a17f-3671ef1517b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 02 21:32:43.227670 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:43.227551 4935 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 21:32:43.227670 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:43.227610 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" podUID="9f07e6bf-04ee-4b56-b10b-ee78424e956c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 21:32:44.180302 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:44.180261 4935 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 02 21:32:44.180879 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:44.180844 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" podUID="f896b6d0-edd2-4201-a17f-3671ef1517b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 02 21:32:44.232157 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:44.225358 4935 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 21:32:44.232157 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:44.225420 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" podUID="9f07e6bf-04ee-4b56-b10b-ee78424e956c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 21:32:45.176066 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:45.176022 4935 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 02 21:32:45.176412 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:45.176380 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" podUID="f896b6d0-edd2-4201-a17f-3671ef1517b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 02 21:32:45.228358 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:45.228322 4935 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 21:32:45.228964 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:45.228932 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" podUID="9f07e6bf-04ee-4b56-b10b-ee78424e956c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 21:32:45.394167 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:45.391748 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:46.175155 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:46.171501 4935 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 02 21:32:46.175155 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:46.171562 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" podUID="f896b6d0-edd2-4201-a17f-3671ef1517b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 02 21:32:46.229154 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:46.228479 4935 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 21:32:46.229154 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:46.228538 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" podUID="9f07e6bf-04ee-4b56-b10b-ee78424e956c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 21:32:47.176264 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:47.176224 4935 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 02 21:32:47.176586 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:47.176550 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" podUID="f896b6d0-edd2-4201-a17f-3671ef1517b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 02 21:32:47.246592 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:47.246546 4935 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 21:32:47.247203 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:47.247165 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" podUID="9f07e6bf-04ee-4b56-b10b-ee78424e956c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 21:32:48.234720 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:48.234680 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:48.239009 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:48.236284 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:48.239366 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:48.239341 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:32:48.266410 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:48.266366 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:48.267107 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:48.267087 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:48.268323 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:48.268301 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:32:48.441045 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:48.440972 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" podStartSLOduration=8.440953591 podStartE2EDuration="8.440953591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:32:42.235859471 +0000 UTC m=+1454.675434196" watchObservedRunningTime="2026-11-02 21:32:48.440953591 +0000 UTC m=+1460.880528314" Nov 02 21:32:50.383689 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:50.383657 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:50.956385 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:50.956329 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:51.167553 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:51.167501 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:51.678679 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:51.678640 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:52.213336 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:52.213298 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:56.122408 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:56.122368 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:56.329593 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:56.329560 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:58.469929 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:32:58.469892 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:32:59.284974 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:59.284934 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:59.524635 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:59.524575 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:32:59.609507 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:59.609417 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9"] Nov 02 21:32:59.616802 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:59.616763 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9" Nov 02 21:32:59.661557 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:59.661514 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp6j2\" (UniqueName: \"kubernetes.io/projected/50e8e193-d60a-4ce2-add8-4e788605bd23-kube-api-access-qp6j2\") pod \"payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9\" (UID: \"50e8e193-d60a-4ce2-add8-4e788605bd23\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9" Nov 02 21:32:59.661746 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:59.661583 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50e8e193-d60a-4ce2-add8-4e788605bd23-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9\" (UID: \"50e8e193-d60a-4ce2-add8-4e788605bd23\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9" Nov 02 21:32:59.661868 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:59.661835 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9"] Nov 02 21:32:59.704620 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:59.704566 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-bd96ef\"" Nov 02 21:32:59.705142 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:59.705095 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-bd96ef-dockercfg-txxf8\"" Nov 02 21:32:59.786156 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:59.785907 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp6j2\" (UniqueName: \"kubernetes.io/projected/50e8e193-d60a-4ce2-add8-4e788605bd23-kube-api-access-qp6j2\") pod \"payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9\" (UID: \"50e8e193-d60a-4ce2-add8-4e788605bd23\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9" Nov 02 21:32:59.786156 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:59.786001 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50e8e193-d60a-4ce2-add8-4e788605bd23-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9\" (UID: \"50e8e193-d60a-4ce2-add8-4e788605bd23\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9" Nov 02 21:32:59.800160 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:59.786996 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50e8e193-d60a-4ce2-add8-4e788605bd23-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9\" (UID: \"50e8e193-d60a-4ce2-add8-4e788605bd23\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9" Nov 02 21:32:59.884529 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:59.882042 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp6j2\" (UniqueName: \"kubernetes.io/projected/50e8e193-d60a-4ce2-add8-4e788605bd23-kube-api-access-qp6j2\") pod \"payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9\" (UID: \"50e8e193-d60a-4ce2-add8-4e788605bd23\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9" Nov 02 21:32:59.933373 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:32:59.933333 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9" Nov 02 21:33:00.314187 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:00.314141 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9"] Nov 02 21:33:00.315919 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:33:00.315875 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e8e193_d60a_4ce2_add8_4e788605bd23.slice/crio-1a09852ec154735e6bb3ff6567d7b4261b731c81957c0e2e2e8332f2239c0108 WatchSource:0}: Error finding container 1a09852ec154735e6bb3ff6567d7b4261b731c81957c0e2e2e8332f2239c0108: Status 404 returned error can't find the container with id 1a09852ec154735e6bb3ff6567d7b4261b731c81957c0e2e2e8332f2239c0108 Nov 02 21:33:01.215312 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:01.215273 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9" event={"ID":"50e8e193-d60a-4ce2-add8-4e788605bd23","Type":"ContainerStarted","Data":"1a09852ec154735e6bb3ff6567d7b4261b731c81957c0e2e2e8332f2239c0108"} Nov 02 21:33:01.728249 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:01.727332 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:04.245049 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:04.244997 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9" event={"ID":"50e8e193-d60a-4ce2-add8-4e788605bd23","Type":"ContainerStarted","Data":"e44a07b6e078b8cd91d8361b825c551d7b3798ee2c5f3abd3299428fb98b9859"} Nov 02 21:33:04.245569 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:04.245106 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9" Nov 02 21:33:04.356679 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:04.356611 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9" podStartSLOduration=2.400038752 podStartE2EDuration="5.356590725s" totalImagesPullingTime="2.956551973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:04.351008911 +0000 UTC m=+1476.790583636" watchObservedRunningTime="2026-11-02 21:33:04.356590725 +0000 UTC m=+1476.796165448" Nov 02 21:33:06.834908 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:06.834207 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:09.293695 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.293606 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g"] Nov 02 21:33:09.300021 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.299987 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:09.423174 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.423114 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g"] Nov 02 21:33:09.428061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.426484 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:09.428061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.426535 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:09.428061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.426656 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:09.428061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.426683 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:09.428061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.426756 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:09.428061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.426785 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:09.428061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.426829 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:09.428061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.426875 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:09.428061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.427007 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjxgx\" (UniqueName: \"kubernetes.io/projected/00c2f98e-d274-4560-8f69-ca6e2dd1888d-kube-api-access-wjxgx\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:09.528431 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.528393 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:09.528431 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.528433 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:09.528641 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.528533 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:09.528641 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.528558 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:09.528641 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.528625 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:09.528754 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.528653 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:09.528754 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.528695 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:09.528754 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.528739 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:09.528890 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.528872 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjxgx\" (UniqueName: \"kubernetes.io/projected/00c2f98e-d274-4560-8f69-ca6e2dd1888d-kube-api-access-wjxgx\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:09.528994 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.528873 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:09.529344 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.529320 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:09.529456 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.529415 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:09.529522 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.529499 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:09.529565 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.529499 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:09.530926 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.530898 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:09.531319 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.531299 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:09.587024 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.586945 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:09.589370 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.589330 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjxgx\" (UniqueName: \"kubernetes.io/projected/00c2f98e-d274-4560-8f69-ca6e2dd1888d-kube-api-access-wjxgx\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:09.623778 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:09.623745 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:10.062714 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:10.062680 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g"] Nov 02 21:33:10.063753 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:33:10.063711 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c2f98e_d274_4560_8f69_ca6e2dd1888d.slice/crio-4f8f630541fc13cf4a1987215d11fd5cbd9a08c087e1a7b074aa1b9fd9f1c2b5 WatchSource:0}: Error finding container 4f8f630541fc13cf4a1987215d11fd5cbd9a08c087e1a7b074aa1b9fd9f1c2b5: Status 404 returned error can't find the container with id 4f8f630541fc13cf4a1987215d11fd5cbd9a08c087e1a7b074aa1b9fd9f1c2b5 Nov 02 21:33:10.067002 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:10.066965 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:33:10.067123 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:10.067034 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:33:10.067123 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:10.067063 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:33:10.297766 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:10.297734 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" event={"ID":"00c2f98e-d274-4560-8f69-ca6e2dd1888d","Type":"ContainerStarted","Data":"66a939f5b5b849eb65ada94d6bc78b5c828e69ba5e0509c8bed9966f01bd24c8"} Nov 02 21:33:10.298235 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:10.297788 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" event={"ID":"00c2f98e-d274-4560-8f69-ca6e2dd1888d","Type":"ContainerStarted","Data":"4f8f630541fc13cf4a1987215d11fd5cbd9a08c087e1a7b074aa1b9fd9f1c2b5"} Nov 02 21:33:10.410709 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:10.410586 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" podStartSLOduration=1.41056346 podStartE2EDuration="1.41056346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:10.409989979 +0000 UTC m=+1482.849564703" watchObservedRunningTime="2026-11-02 21:33:10.41056346 +0000 UTC m=+1482.850138182" Nov 02 21:33:10.625824 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:10.625783 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:11.642691 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:11.642276 4935 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 02 21:33:11.642691 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:11.642344 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" podUID="00c2f98e-d274-4560-8f69-ca6e2dd1888d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 02 21:33:12.146307 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:12.146099 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:12.631156 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:12.627488 4935 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 02 21:33:12.631156 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:12.627553 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" podUID="00c2f98e-d274-4560-8f69-ca6e2dd1888d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 02 21:33:13.628168 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:13.627474 4935 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 02 21:33:13.628168 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:13.627547 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" podUID="00c2f98e-d274-4560-8f69-ca6e2dd1888d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 02 21:33:14.649554 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:14.649373 4935 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 02 21:33:14.649554 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:14.649455 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" podUID="00c2f98e-d274-4560-8f69-ca6e2dd1888d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 02 21:33:15.256089 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:15.256055 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9" Nov 02 21:33:15.639518 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:15.639477 4935 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 02 21:33:15.639781 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:15.639754 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" podUID="00c2f98e-d274-4560-8f69-ca6e2dd1888d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 02 21:33:16.625967 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:16.625927 4935 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 02 21:33:16.626405 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:16.625995 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" podUID="00c2f98e-d274-4560-8f69-ca6e2dd1888d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 02 21:33:17.227121 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:17.227080 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:17.629582 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:17.629552 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:17.629971 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:17.629921 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:17.630640 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:17.630618 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:33:21.319680 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:21.319638 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:21.546204 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:21.546169 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:22.315391 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:22.315347 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:25.819122 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:25.819090 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:26.862334 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:26.862289 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:26.986746 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:26.986699 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g"] Nov 02 21:33:26.991166 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:26.991142 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g" Nov 02 21:33:27.022542 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:27.022508 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d34e34f3-77b3-433c-8e39-07d64e93fb32-metrics-certs\") pod \"maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g\" (UID: \"d34e34f3-77b3-433c-8e39-07d64e93fb32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g" Nov 02 21:33:27.022757 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:27.022559 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d34e34f3-77b3-433c-8e39-07d64e93fb32-maas-api-tls\") pod \"maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g\" (UID: \"d34e34f3-77b3-433c-8e39-07d64e93fb32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g" Nov 02 21:33:27.022757 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:27.022668 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d34e34f3-77b3-433c-8e39-07d64e93fb32-openshift-service-ca\") pod \"maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g\" (UID: \"d34e34f3-77b3-433c-8e39-07d64e93fb32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g" Nov 02 21:33:27.022757 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:27.022706 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85mrf\" (UniqueName: \"kubernetes.io/projected/d34e34f3-77b3-433c-8e39-07d64e93fb32-kube-api-access-85mrf\") pod \"maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g\" (UID: \"d34e34f3-77b3-433c-8e39-07d64e93fb32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g" Nov 02 21:33:27.086275 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:27.086240 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-54a4b2-serving-cert\"" Nov 02 21:33:27.090718 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:27.090676 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g"] Nov 02 21:33:27.123899 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:27.123804 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d34e34f3-77b3-433c-8e39-07d64e93fb32-metrics-certs\") pod \"maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g\" (UID: \"d34e34f3-77b3-433c-8e39-07d64e93fb32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g" Nov 02 21:33:27.123899 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:27.123863 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d34e34f3-77b3-433c-8e39-07d64e93fb32-maas-api-tls\") pod \"maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g\" (UID: \"d34e34f3-77b3-433c-8e39-07d64e93fb32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g" Nov 02 21:33:27.123899 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:27.123899 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d34e34f3-77b3-433c-8e39-07d64e93fb32-openshift-service-ca\") pod \"maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g\" (UID: \"d34e34f3-77b3-433c-8e39-07d64e93fb32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g" Nov 02 21:33:27.124269 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:27.123925 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85mrf\" (UniqueName: \"kubernetes.io/projected/d34e34f3-77b3-433c-8e39-07d64e93fb32-kube-api-access-85mrf\") pod \"maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g\" (UID: \"d34e34f3-77b3-433c-8e39-07d64e93fb32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g" Nov 02 21:33:27.124269 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:33:27.124030 4935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-serving-cert: secret "maas-api-e2e-api-b-54a4b2-serving-cert" not found Nov 02 21:33:27.124269 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:33:27.124120 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d34e34f3-77b3-433c-8e39-07d64e93fb32-maas-api-tls podName:d34e34f3-77b3-433c-8e39-07d64e93fb32 nodeName:}" failed. No retries permitted until 2026-11-02 21:33:27.624099448 +0000 UTC m=+1500.063674172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d34e34f3-77b3-433c-8e39-07d64e93fb32-maas-api-tls") pod "maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g" (UID: "d34e34f3-77b3-433c-8e39-07d64e93fb32") : secret "maas-api-e2e-api-b-54a4b2-serving-cert" not found Nov 02 21:33:27.124670 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:27.124649 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d34e34f3-77b3-433c-8e39-07d64e93fb32-openshift-service-ca\") pod \"maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g\" (UID: \"d34e34f3-77b3-433c-8e39-07d64e93fb32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g" Nov 02 21:33:27.126494 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:27.126468 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d34e34f3-77b3-433c-8e39-07d64e93fb32-metrics-certs\") pod \"maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g\" (UID: \"d34e34f3-77b3-433c-8e39-07d64e93fb32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g" Nov 02 21:33:27.177914 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:27.177878 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85mrf\" (UniqueName: \"kubernetes.io/projected/d34e34f3-77b3-433c-8e39-07d64e93fb32-kube-api-access-85mrf\") pod \"maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g\" (UID: \"d34e34f3-77b3-433c-8e39-07d64e93fb32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g" Nov 02 21:33:27.274078 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:27.274039 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:27.583815 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:27.583771 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:27.635088 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:27.635047 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d34e34f3-77b3-433c-8e39-07d64e93fb32-maas-api-tls\") pod \"maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g\" (UID: \"d34e34f3-77b3-433c-8e39-07d64e93fb32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g" Nov 02 21:33:27.659157 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:27.654826 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d34e34f3-77b3-433c-8e39-07d64e93fb32-maas-api-tls\") pod \"maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g\" (UID: \"d34e34f3-77b3-433c-8e39-07d64e93fb32\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g" Nov 02 21:33:27.800913 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:27.800858 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:27.906046 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:27.905946 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g" Nov 02 21:33:28.085820 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:28.085787 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 21:33:28.091521 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:28.091469 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 21:33:28.095116 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:28.095087 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 21:33:28.103016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:28.102997 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 21:33:28.296663 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:33:28.296633 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd34e34f3_77b3_433c_8e39_07d64e93fb32.slice/crio-11cb25979473d1dfb1aacaef57c632da77619879590d806611f89b0b7aff8fbc WatchSource:0}: Error finding container 11cb25979473d1dfb1aacaef57c632da77619879590d806611f89b0b7aff8fbc: Status 404 returned error can't find the container with id 11cb25979473d1dfb1aacaef57c632da77619879590d806611f89b0b7aff8fbc Nov 02 21:33:28.297120 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:28.297094 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g"] Nov 02 21:33:28.487485 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:28.487378 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g" event={"ID":"d34e34f3-77b3-433c-8e39-07d64e93fb32","Type":"ContainerStarted","Data":"11cb25979473d1dfb1aacaef57c632da77619879590d806611f89b0b7aff8fbc"} Nov 02 21:33:28.518231 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:33:28.514189 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:33:29.495898 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:29.495857 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g" event={"ID":"d34e34f3-77b3-433c-8e39-07d64e93fb32","Type":"ContainerStarted","Data":"3f978fba62849a05db0bb0e6f515195e092d4dfc5fe6461201899ae43818d806"} Nov 02 21:33:29.496327 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:29.495926 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g" Nov 02 21:33:29.606891 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:29.606834 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g" podStartSLOduration=2.655824227 podStartE2EDuration="3.606814301s" totalImagesPullingTime="950.990074ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:33:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:29.603532541 +0000 UTC m=+1502.043107266" watchObservedRunningTime="2026-11-02 21:33:29.606814301 +0000 UTC m=+1502.046389026" Nov 02 21:33:31.838943 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:31.838899 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:32.049843 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:32.049803 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:32.802140 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:32.802088 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:34.435339 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:34.435307 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:34.646907 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:34.646873 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:35.488772 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:35.488738 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:35.560666 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:35.560403 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g" Nov 02 21:33:35.800690 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:35.800651 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:36.134767 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:36.134676 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d"] Nov 02 21:33:36.140775 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:36.140749 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d" Nov 02 21:33:36.189280 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:36.189243 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d"] Nov 02 21:33:36.189590 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:36.189563 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-682116\"" Nov 02 21:33:36.247154 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:36.231249 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-682116-dockercfg-9qln9\"" Nov 02 21:33:36.275076 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:36.275002 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4efcdacc-c172-4f9a-b877-3b0a510252de-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d\" (UID: \"4efcdacc-c172-4f9a-b877-3b0a510252de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d" Nov 02 21:33:36.275323 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:36.275119 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v9mx\" (UniqueName: \"kubernetes.io/projected/4efcdacc-c172-4f9a-b877-3b0a510252de-kube-api-access-6v9mx\") pod \"payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d\" (UID: \"4efcdacc-c172-4f9a-b877-3b0a510252de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d" Nov 02 21:33:36.376381 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:36.376346 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4efcdacc-c172-4f9a-b877-3b0a510252de-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d\" (UID: \"4efcdacc-c172-4f9a-b877-3b0a510252de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d" Nov 02 21:33:36.376381 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:36.376385 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v9mx\" (UniqueName: \"kubernetes.io/projected/4efcdacc-c172-4f9a-b877-3b0a510252de-kube-api-access-6v9mx\") pod \"payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d\" (UID: \"4efcdacc-c172-4f9a-b877-3b0a510252de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d" Nov 02 21:33:36.377089 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:36.377066 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4efcdacc-c172-4f9a-b877-3b0a510252de-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d\" (UID: \"4efcdacc-c172-4f9a-b877-3b0a510252de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d" Nov 02 21:33:36.430105 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:36.430028 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v9mx\" (UniqueName: \"kubernetes.io/projected/4efcdacc-c172-4f9a-b877-3b0a510252de-kube-api-access-6v9mx\") pod \"payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d\" (UID: \"4efcdacc-c172-4f9a-b877-3b0a510252de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d" Nov 02 21:33:36.459046 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:36.459016 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d" Nov 02 21:33:36.839035 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:36.839002 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d"] Nov 02 21:33:36.840054 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:33:36.840027 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4efcdacc_c172_4f9a_b877_3b0a510252de.slice/crio-cfcd5c1fc21dbfc109ffb24bf5f98a519c61524a1313779972ae659098017732 WatchSource:0}: Error finding container cfcd5c1fc21dbfc109ffb24bf5f98a519c61524a1313779972ae659098017732: Status 404 returned error can't find the container with id cfcd5c1fc21dbfc109ffb24bf5f98a519c61524a1313779972ae659098017732 Nov 02 21:33:37.589989 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:37.589948 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d" event={"ID":"4efcdacc-c172-4f9a-b877-3b0a510252de","Type":"ContainerStarted","Data":"3724f4c45127d5a011ea69697b363b4b24a7ac03c3b08abcc68b64ce5b085b38"} Nov 02 21:33:37.590201 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:37.590011 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d" event={"ID":"4efcdacc-c172-4f9a-b877-3b0a510252de","Type":"ContainerStarted","Data":"cfcd5c1fc21dbfc109ffb24bf5f98a519c61524a1313779972ae659098017732"} Nov 02 21:33:37.590201 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:37.590035 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d" Nov 02 21:33:37.698886 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:37.698834 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d" podStartSLOduration=1.698817354 podStartE2EDuration="1.698817354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:37.698515065 +0000 UTC m=+1510.138089786" watchObservedRunningTime="2026-11-02 21:33:37.698817354 +0000 UTC m=+1510.138392078" Nov 02 21:33:38.120991 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:38.120931 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:38.584926 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:38.584892 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7471-dab6c6"] Nov 02 21:33:38.588346 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:38.588323 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7471-dab6c6" Nov 02 21:33:38.640932 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:38.640900 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7471-dab6c6"] Nov 02 21:33:38.681752 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:38.681715 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5v2nf\"" Nov 02 21:33:38.706310 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:38.706279 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xz8h\" (UniqueName: \"kubernetes.io/projected/ba474111-ffaf-458a-9a78-0625d63ff967-kube-api-access-6xz8h\") pod \"test-curl-7471-dab6c6\" (UID: \"ba474111-ffaf-458a-9a78-0625d63ff967\") " pod="opendatahub/test-curl-7471-dab6c6" Nov 02 21:33:38.807584 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:38.807546 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xz8h\" (UniqueName: \"kubernetes.io/projected/ba474111-ffaf-458a-9a78-0625d63ff967-kube-api-access-6xz8h\") pod \"test-curl-7471-dab6c6\" (UID: \"ba474111-ffaf-458a-9a78-0625d63ff967\") " pod="opendatahub/test-curl-7471-dab6c6" Nov 02 21:33:38.863181 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:38.863087 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xz8h\" (UniqueName: \"kubernetes.io/projected/ba474111-ffaf-458a-9a78-0625d63ff967-kube-api-access-6xz8h\") pod \"test-curl-7471-dab6c6\" (UID: \"ba474111-ffaf-458a-9a78-0625d63ff967\") " pod="opendatahub/test-curl-7471-dab6c6" Nov 02 21:33:38.905154 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:38.905112 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7471-dab6c6" Nov 02 21:33:39.287047 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:39.287010 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7471-dab6c6"] Nov 02 21:33:39.287398 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:33:39.287300 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba474111_ffaf_458a_9a78_0625d63ff967.slice/crio-902ae28c45d70dc8bfd80706c90b70cace9b77d03be7420c3a8ce371bf57c846 WatchSource:0}: Error finding container 902ae28c45d70dc8bfd80706c90b70cace9b77d03be7420c3a8ce371bf57c846: Status 404 returned error can't find the container with id 902ae28c45d70dc8bfd80706c90b70cace9b77d03be7420c3a8ce371bf57c846 Nov 02 21:33:39.619893 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:39.619805 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7471-dab6c6" event={"ID":"ba474111-ffaf-458a-9a78-0625d63ff967","Type":"ContainerStarted","Data":"902ae28c45d70dc8bfd80706c90b70cace9b77d03be7420c3a8ce371bf57c846"} Nov 02 21:33:42.650075 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:42.649973 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7471-dab6c6" event={"ID":"ba474111-ffaf-458a-9a78-0625d63ff967","Type":"ContainerStarted","Data":"ec377f84e0b2f0f646df7ff237f9723186228647af16020d806331bfbb3201d9"} Nov 02 21:33:42.757934 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:42.757881 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7471-dab6c6" podStartSLOduration=1.779200369 podStartE2EDuration="4.757865723s" totalImagesPullingTime="2.978665354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:42.756405932 +0000 UTC m=+1515.195980656" watchObservedRunningTime="2026-11-02 21:33:42.757865723 +0000 UTC m=+1515.197440444" Nov 02 21:33:43.153034 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:43.153001 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:44.268089 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:44.268040 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7471-dab6c6"] Nov 02 21:33:44.272643 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:44.272601 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7471-dab6c6"] Nov 02 21:33:44.671721 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:44.671629 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7471-dab6c6" podUID="ba474111-ffaf-458a-9a78-0625d63ff967" containerName="test-curl-7471-dab6c6" containerID="cri-o://ec377f84e0b2f0f646df7ff237f9723186228647af16020d806331bfbb3201d9" gracePeriod=2 Nov 02 21:33:44.720203 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:33:44.720162 4935 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7471-dab6c6\" is forbidden: User \"system:node:ip-10-0-128-89.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-89.us-west-2.compute.internal' and this object" podUID="ba474111-ffaf-458a-9a78-0625d63ff967" pod="opendatahub/test-curl-7471-dab6c6" Nov 02 21:33:45.289885 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:45.289848 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:45.570954 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:45.570879 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:46.954372 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:46.954302 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7471-dab6c6" Nov 02 21:33:47.020609 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:47.020572 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba474111-ffaf-458a-9a78-0625d63ff967-kube-api-access-6xz8h\" (UniqueName: \"kubernetes.io/projected/ba474111-ffaf-458a-9a78-0625d63ff967-kube-api-access-6xz8h\") pod \"ba474111-ffaf-458a-9a78-0625d63ff967\" (UID: \"ba474111-ffaf-458a-9a78-0625d63ff967\") " Nov 02 21:33:47.022710 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:47.022681 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba474111-ffaf-458a-9a78-0625d63ff967-kube-api-access-6xz8h" pod "ba474111-ffaf-458a-9a78-0625d63ff967" (UID: "ba474111-ffaf-458a-9a78-0625d63ff967"). InnerVolumeSpecName "kube-api-access-6xz8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:33:47.122611 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:47.122577 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xz8h\" (UniqueName: \"kubernetes.io/projected/ba474111-ffaf-458a-9a78-0625d63ff967-kube-api-access-6xz8h\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:33:47.694180 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:47.694112 4935 generic.go:356] "Generic (PLEG): container finished" podID="ba474111-ffaf-458a-9a78-0625d63ff967" containerID="ec377f84e0b2f0f646df7ff237f9723186228647af16020d806331bfbb3201d9" exitCode=137 Nov 02 21:33:47.694374 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:47.694219 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7471-dab6c6" Nov 02 21:33:47.694374 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:47.694220 4935 scope.go:122] "RemoveContainer" containerID="ec377f84e0b2f0f646df7ff237f9723186228647af16020d806331bfbb3201d9" Nov 02 21:33:47.706366 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:47.706335 4935 scope.go:122] "RemoveContainer" containerID="ec377f84e0b2f0f646df7ff237f9723186228647af16020d806331bfbb3201d9" Nov 02 21:33:47.706665 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:33:47.706644 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec377f84e0b2f0f646df7ff237f9723186228647af16020d806331bfbb3201d9\": container with ID starting with ec377f84e0b2f0f646df7ff237f9723186228647af16020d806331bfbb3201d9 not found: ID does not exist" containerID="ec377f84e0b2f0f646df7ff237f9723186228647af16020d806331bfbb3201d9" Nov 02 21:33:47.706731 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:47.706674 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec377f84e0b2f0f646df7ff237f9723186228647af16020d806331bfbb3201d9"} err="failed to get container status \"ec377f84e0b2f0f646df7ff237f9723186228647af16020d806331bfbb3201d9\": rpc error: code = NotFound desc = could not find container \"ec377f84e0b2f0f646df7ff237f9723186228647af16020d806331bfbb3201d9\": container with ID starting with ec377f84e0b2f0f646df7ff237f9723186228647af16020d806331bfbb3201d9 not found: ID does not exist" Nov 02 21:33:47.752472 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:33:47.752429 4935 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7471-dab6c6\" is forbidden: User \"system:node:ip-10-0-128-89.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-89.us-west-2.compute.internal' and this object" podUID="ba474111-ffaf-458a-9a78-0625d63ff967" pod="opendatahub/test-curl-7471-dab6c6" Nov 02 21:33:47.780766 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:47.780728 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba474111-ffaf-458a-9a78-0625d63ff967" path="/var/lib/kubelet/pods/ba474111-ffaf-458a-9a78-0625d63ff967/volumes" Nov 02 21:33:47.821246 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:33:47.821205 4935 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7471-dab6c6\" is forbidden: User \"system:node:ip-10-0-128-89.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-89.us-west-2.compute.internal' and this object" podUID="ba474111-ffaf-458a-9a78-0625d63ff967" pod="opendatahub/test-curl-7471-dab6c6" Nov 02 21:33:48.470154 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:48.469983 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:48.606044 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:48.605716 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d" Nov 02 21:33:48.684853 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:48.684806 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:48.890672 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:48.890640 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:54.081345 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:54.081270 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:56.991966 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:56.991932 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:57.198415 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:57.198369 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:33:58.480857 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:33:58.480819 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:33:59.223803 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:33:59.223770 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:04.219872 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:04.219823 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:06.826388 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:06.826356 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:09.233672 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:09.233630 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:13.149088 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:13.149053 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:14.338947 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:14.338911 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:19.330013 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:19.329973 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:24.305582 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:24.305539 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:26.142626 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:26.142581 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:28.470163 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:34:28.470046 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:34:28.989986 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:28.989951 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq"] Nov 02 21:34:28.990597 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:28.990573 4935 cpu_manager.go:414] "containerMap: removing container" podUID="ba474111-ffaf-458a-9a78-0625d63ff967" containerName="test-curl-7471-dab6c6" Nov 02 21:34:28.990597 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:28.990595 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba474111-ffaf-458a-9a78-0625d63ff967" containerName="test-curl-7471-dab6c6" Nov 02 21:34:28.990811 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:28.990799 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba474111-ffaf-458a-9a78-0625d63ff967" containerName="test-curl-7471-dab6c6" Nov 02 21:34:28.996588 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:28.996570 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq" Nov 02 21:34:29.053759 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:29.053720 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq"] Nov 02 21:34:29.056803 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:29.056778 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-682116-serving-cert\"" Nov 02 21:34:29.123750 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:29.123715 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa406b90-6571-4f44-a9ba-b3299863e736-maas-api-tls\") pod \"maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq\" (UID: \"aa406b90-6571-4f44-a9ba-b3299863e736\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq" Nov 02 21:34:29.123939 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:29.123839 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7vnl\" (UniqueName: \"kubernetes.io/projected/aa406b90-6571-4f44-a9ba-b3299863e736-kube-api-access-t7vnl\") pod \"maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq\" (UID: \"aa406b90-6571-4f44-a9ba-b3299863e736\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq" Nov 02 21:34:29.124022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:29.123952 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa406b90-6571-4f44-a9ba-b3299863e736-metrics-certs\") pod \"maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq\" (UID: \"aa406b90-6571-4f44-a9ba-b3299863e736\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq" Nov 02 21:34:29.124113 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:29.124085 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa406b90-6571-4f44-a9ba-b3299863e736-openshift-service-ca\") pod \"maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq\" (UID: \"aa406b90-6571-4f44-a9ba-b3299863e736\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq" Nov 02 21:34:29.225477 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:29.225432 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7vnl\" (UniqueName: \"kubernetes.io/projected/aa406b90-6571-4f44-a9ba-b3299863e736-kube-api-access-t7vnl\") pod \"maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq\" (UID: \"aa406b90-6571-4f44-a9ba-b3299863e736\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq" Nov 02 21:34:29.225677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:29.225534 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa406b90-6571-4f44-a9ba-b3299863e736-metrics-certs\") pod \"maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq\" (UID: \"aa406b90-6571-4f44-a9ba-b3299863e736\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq" Nov 02 21:34:29.225677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:29.225631 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa406b90-6571-4f44-a9ba-b3299863e736-openshift-service-ca\") pod \"maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq\" (UID: \"aa406b90-6571-4f44-a9ba-b3299863e736\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq" Nov 02 21:34:29.225792 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:29.225775 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa406b90-6571-4f44-a9ba-b3299863e736-maas-api-tls\") pod \"maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq\" (UID: \"aa406b90-6571-4f44-a9ba-b3299863e736\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq" Nov 02 21:34:29.225939 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:34:29.225915 4935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-682116-serving-cert: secret "maas-api-e2e-api-a-682116-serving-cert" not found Nov 02 21:34:29.226003 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:34:29.225991 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa406b90-6571-4f44-a9ba-b3299863e736-maas-api-tls podName:aa406b90-6571-4f44-a9ba-b3299863e736 nodeName:}" failed. No retries permitted until 2026-11-02 21:34:29.725969056 +0000 UTC m=+1562.165543778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aa406b90-6571-4f44-a9ba-b3299863e736-maas-api-tls") pod "maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq" (UID: "aa406b90-6571-4f44-a9ba-b3299863e736") : secret "maas-api-e2e-api-a-682116-serving-cert" not found Nov 02 21:34:29.226435 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:29.226415 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa406b90-6571-4f44-a9ba-b3299863e736-openshift-service-ca\") pod \"maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq\" (UID: \"aa406b90-6571-4f44-a9ba-b3299863e736\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq" Nov 02 21:34:29.228698 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:29.228671 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa406b90-6571-4f44-a9ba-b3299863e736-metrics-certs\") pod \"maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq\" (UID: \"aa406b90-6571-4f44-a9ba-b3299863e736\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq" Nov 02 21:34:29.259387 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:29.259310 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:29.281236 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:29.281215 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7vnl\" (UniqueName: \"kubernetes.io/projected/aa406b90-6571-4f44-a9ba-b3299863e736-kube-api-access-t7vnl\") pod \"maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq\" (UID: \"aa406b90-6571-4f44-a9ba-b3299863e736\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq" Nov 02 21:34:29.513208 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:29.513172 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:29.719622 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:29.719585 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:29.735482 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:29.735451 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa406b90-6571-4f44-a9ba-b3299863e736-maas-api-tls\") pod \"maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq\" (UID: \"aa406b90-6571-4f44-a9ba-b3299863e736\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq" Nov 02 21:34:29.738041 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:29.738010 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa406b90-6571-4f44-a9ba-b3299863e736-maas-api-tls\") pod \"maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq\" (UID: \"aa406b90-6571-4f44-a9ba-b3299863e736\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq" Nov 02 21:34:29.910665 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:29.910582 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq" Nov 02 21:34:30.292068 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:30.292039 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq"] Nov 02 21:34:30.293038 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:34:30.293010 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa406b90_6571_4f44_a9ba_b3299863e736.slice/crio-88456a06d6a571e3585b72fd79fd2750261dcf376417ff7dda40e5f9c653e4bd WatchSource:0}: Error finding container 88456a06d6a571e3585b72fd79fd2750261dcf376417ff7dda40e5f9c653e4bd: Status 404 returned error can't find the container with id 88456a06d6a571e3585b72fd79fd2750261dcf376417ff7dda40e5f9c653e4bd Nov 02 21:34:31.071116 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:31.071072 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq" event={"ID":"aa406b90-6571-4f44-a9ba-b3299863e736","Type":"ContainerStarted","Data":"88456a06d6a571e3585b72fd79fd2750261dcf376417ff7dda40e5f9c653e4bd"} Nov 02 21:34:32.081538 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:32.081502 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq" event={"ID":"aa406b90-6571-4f44-a9ba-b3299863e736","Type":"ContainerStarted","Data":"a8e3baecc27c1cd088d455a47b3231c90ef456655e1057400cd1d77c7bf9f02c"} Nov 02 21:34:32.081902 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:32.081651 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq" Nov 02 21:34:32.191405 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:32.191358 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq" podStartSLOduration=3.243692315 podStartE2EDuration="4.191342245s" totalImagesPullingTime="947.64993ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:32.188786672 +0000 UTC m=+1564.628361395" watchObservedRunningTime="2026-11-02 21:34:32.191342245 +0000 UTC m=+1564.630916967" Nov 02 21:34:34.112705 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:34.112666 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:34.802730 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:34.802691 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:36.668265 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:36.668230 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:38.094741 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:38.094666 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq" Nov 02 21:34:39.856588 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:39.856549 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:42.651484 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:42.651433 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:43.598898 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:43.598856 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:43.812660 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:43.812625 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:45.116917 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:45.116871 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:47.002871 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:47.002821 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:48.371790 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:48.371734 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:50.485370 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:50.485334 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:53.411841 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:53.411806 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:55.556955 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:55.556922 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:56.143545 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:34:56.143508 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:34:58.471548 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:34:58.471503 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:35:01.469018 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:01.468981 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:01.775593 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:01.773247 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:02.254794 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:02.254687 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:02.479228 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:02.479125 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:06.534796 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:06.534763 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8"] Nov 02 21:35:06.535295 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:06.535025 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8" podUID="6a33cd12-7bf8-4b6a-af93-201a73a69240" containerName="maas-api" containerID="cri-o://0a43f4d59b80875877f06d4618048067c99dbc7e8e5a071a8e472f19f436af14" gracePeriod=30 Nov 02 21:35:06.710856 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:06.710816 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:06.886836 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:06.886793 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8" Nov 02 21:35:06.980069 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:06.978949 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a33cd12-7bf8-4b6a-af93-201a73a69240-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a33cd12-7bf8-4b6a-af93-201a73a69240-maas-api-tls\") pod \"6a33cd12-7bf8-4b6a-af93-201a73a69240\" (UID: \"6a33cd12-7bf8-4b6a-af93-201a73a69240\") " Nov 02 21:35:06.980069 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:06.978995 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a33cd12-7bf8-4b6a-af93-201a73a69240-kube-api-access-vkqqx\" (UniqueName: \"kubernetes.io/projected/6a33cd12-7bf8-4b6a-af93-201a73a69240-kube-api-access-vkqqx\") pod \"6a33cd12-7bf8-4b6a-af93-201a73a69240\" (UID: \"6a33cd12-7bf8-4b6a-af93-201a73a69240\") " Nov 02 21:35:06.980069 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:06.979067 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a33cd12-7bf8-4b6a-af93-201a73a69240-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a33cd12-7bf8-4b6a-af93-201a73a69240-openshift-service-ca\") pod \"6a33cd12-7bf8-4b6a-af93-201a73a69240\" (UID: \"6a33cd12-7bf8-4b6a-af93-201a73a69240\") " Nov 02 21:35:06.980069 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:06.979102 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a33cd12-7bf8-4b6a-af93-201a73a69240-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a33cd12-7bf8-4b6a-af93-201a73a69240-metrics-certs\") pod \"6a33cd12-7bf8-4b6a-af93-201a73a69240\" (UID: \"6a33cd12-7bf8-4b6a-af93-201a73a69240\") " Nov 02 21:35:06.983314 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:06.983241 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a33cd12-7bf8-4b6a-af93-201a73a69240-openshift-service-ca" pod "6a33cd12-7bf8-4b6a-af93-201a73a69240" (UID: "6a33cd12-7bf8-4b6a-af93-201a73a69240"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:06.997154 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:06.988288 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a33cd12-7bf8-4b6a-af93-201a73a69240-kube-api-access-vkqqx" pod "6a33cd12-7bf8-4b6a-af93-201a73a69240" (UID: "6a33cd12-7bf8-4b6a-af93-201a73a69240"). InnerVolumeSpecName "kube-api-access-vkqqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:06.997154 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:06.989965 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a33cd12-7bf8-4b6a-af93-201a73a69240-metrics-certs" pod "6a33cd12-7bf8-4b6a-af93-201a73a69240" (UID: "6a33cd12-7bf8-4b6a-af93-201a73a69240"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:06.997154 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:06.995439 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a33cd12-7bf8-4b6a-af93-201a73a69240-maas-api-tls" pod "6a33cd12-7bf8-4b6a-af93-201a73a69240" (UID: "6a33cd12-7bf8-4b6a-af93-201a73a69240"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:07.081490 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:07.081448 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a33cd12-7bf8-4b6a-af93-201a73a69240-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:07.081666 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:07.081493 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkqqx\" (UniqueName: \"kubernetes.io/projected/6a33cd12-7bf8-4b6a-af93-201a73a69240-kube-api-access-vkqqx\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:07.081666 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:07.081511 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a33cd12-7bf8-4b6a-af93-201a73a69240-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:07.081666 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:07.081528 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a33cd12-7bf8-4b6a-af93-201a73a69240-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:07.307715 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:07.307673 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:07.396818 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:07.396778 4935 generic.go:356] "Generic (PLEG): container finished" podID="6a33cd12-7bf8-4b6a-af93-201a73a69240" containerID="0a43f4d59b80875877f06d4618048067c99dbc7e8e5a071a8e472f19f436af14" exitCode=0 Nov 02 21:35:07.397004 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:07.396829 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8" Nov 02 21:35:07.397004 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:07.396873 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8" event={"ID":"6a33cd12-7bf8-4b6a-af93-201a73a69240","Type":"ContainerDied","Data":"0a43f4d59b80875877f06d4618048067c99dbc7e8e5a071a8e472f19f436af14"} Nov 02 21:35:07.397004 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:07.396939 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8" event={"ID":"6a33cd12-7bf8-4b6a-af93-201a73a69240","Type":"ContainerDied","Data":"3080db0bececd131ad8113ab42a2e360d5d6e5b945f29d36ea4ae5ad2dbe1945"} Nov 02 21:35:07.397004 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:07.396968 4935 scope.go:122] "RemoveContainer" containerID="0a43f4d59b80875877f06d4618048067c99dbc7e8e5a071a8e472f19f436af14" Nov 02 21:35:07.409276 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:07.409259 4935 scope.go:122] "RemoveContainer" containerID="0a43f4d59b80875877f06d4618048067c99dbc7e8e5a071a8e472f19f436af14" Nov 02 21:35:07.409564 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:35:07.409542 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a43f4d59b80875877f06d4618048067c99dbc7e8e5a071a8e472f19f436af14\": container with ID starting with 0a43f4d59b80875877f06d4618048067c99dbc7e8e5a071a8e472f19f436af14 not found: ID does not exist" containerID="0a43f4d59b80875877f06d4618048067c99dbc7e8e5a071a8e472f19f436af14" Nov 02 21:35:07.409650 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:07.409571 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a43f4d59b80875877f06d4618048067c99dbc7e8e5a071a8e472f19f436af14"} err="failed to get container status \"0a43f4d59b80875877f06d4618048067c99dbc7e8e5a071a8e472f19f436af14\": rpc error: code = NotFound desc = could not find container \"0a43f4d59b80875877f06d4618048067c99dbc7e8e5a071a8e472f19f436af14\": container with ID starting with 0a43f4d59b80875877f06d4618048067c99dbc7e8e5a071a8e472f19f436af14 not found: ID does not exist" Nov 02 21:35:07.574209 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:07.573264 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8"] Nov 02 21:35:07.651153 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:07.650656 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d58d18-5cbcf4876f-xfkw8"] Nov 02 21:35:07.778979 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:07.778948 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a33cd12-7bf8-4b6a-af93-201a73a69240" path="/var/lib/kubelet/pods/6a33cd12-7bf8-4b6a-af93-201a73a69240/volumes" Nov 02 21:35:08.618710 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:08.618518 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:12.736996 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:12.736901 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:12.961415 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:12.961371 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:13.733029 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:13.732993 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:14.393832 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:14.393782 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:14.970922 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:14.970863 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs"] Nov 02 21:35:14.971492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:14.971454 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" podUID="9f07e6bf-04ee-4b56-b10b-ee78424e956c" containerName="istio-proxy" containerID="cri-o://fa6aaabac3bbb08c0b7f65282f3c1be4763bd6eab6035dd092b1010629bf2488" gracePeriod=30 Nov 02 21:35:14.982929 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:14.982895 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8"] Nov 02 21:35:14.983161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:14.983096 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" podUID="5ba481e7-6f26-4ae3-91cf-0f2acea6fc33" containerName="istio-proxy" containerID="cri-o://e141428f1d65ff4b072fdffb25e7be307644fdeca98631b39e6193f5685ba17f" gracePeriod=30 Nov 02 21:35:15.006119 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:15.006086 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p"] Nov 02 21:35:15.006350 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:15.006313 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" podUID="f896b6d0-edd2-4201-a17f-3671ef1517b7" containerName="istio-proxy" containerID="cri-o://8530b83c064e4c4140f4592d4935aa62c300378971a82fbef3bb6b2d6c2b1f7c" gracePeriod=30 Nov 02 21:35:15.069187 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:15.069060 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:15.099081 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:15.099011 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f750d0c-r5w2v"] Nov 02 21:35:15.147915 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:15.147839 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f750d0c-r5w2v"] Nov 02 21:35:15.779373 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:15.779340 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b" path="/var/lib/kubelet/pods/696dc3c6-32aa-4eb3-984e-cae3a4dc7b3b/volumes" Nov 02 21:35:15.800071 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:15.800037 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:17.658818 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:17.658781 4935 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:35:17.659249 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:17.658859 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" podUID="5ba481e7-6f26-4ae3-91cf-0f2acea6fc33" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:35:18.238021 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:18.237980 4935 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:35:18.238242 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:18.238049 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" podUID="9f07e6bf-04ee-4b56-b10b-ee78424e956c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:35:18.268732 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:18.268695 4935 patch_prober.go:29] interesting pod/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:35:18.268903 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:18.268751 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" podUID="f896b6d0-edd2-4201-a17f-3671ef1517b7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:35:19.265272 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:19.265232 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:20.320688 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.320659 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:35:20.361499 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.361470 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:35:20.363468 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.363447 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:35:20.453774 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.453740 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f896b6d0-edd2-4201-a17f-3671ef1517b7-istio-token\" (UniqueName: \"kubernetes.io/projected/f896b6d0-edd2-4201-a17f-3671ef1517b7-istio-token\") pod \"f896b6d0-edd2-4201-a17f-3671ef1517b7\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " Nov 02 21:35:20.453952 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.453789 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-credential-socket\") pod \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " Nov 02 21:35:20.453952 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.453816 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f896b6d0-edd2-4201-a17f-3671ef1517b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f896b6d0-edd2-4201-a17f-3671ef1517b7-istio-podinfo\") pod \"f896b6d0-edd2-4201-a17f-3671ef1517b7\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " Nov 02 21:35:20.453952 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.453862 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istio-podinfo\") pod \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " Nov 02 21:35:20.453952 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.453885 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istio-token\" (UniqueName: \"kubernetes.io/projected/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istio-token\") pod \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " Nov 02 21:35:20.453952 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.453906 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istio-envoy\") pod \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " Nov 02 21:35:20.453952 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.453950 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istio-envoy\") pod \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " Nov 02 21:35:20.454321 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.454007 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-workload-socket\") pod \"f896b6d0-edd2-4201-a17f-3671ef1517b7\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " Nov 02 21:35:20.454321 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.454034 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-istio-envoy\") pod \"f896b6d0-edd2-4201-a17f-3671ef1517b7\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " Nov 02 21:35:20.454321 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.454065 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-kube-api-access-4cq5b\" (UniqueName: \"kubernetes.io/projected/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-kube-api-access-4cq5b\") pod \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " Nov 02 21:35:20.454321 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.454073 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-credential-socket" pod "9f07e6bf-04ee-4b56-b10b-ee78424e956c" (UID: "9f07e6bf-04ee-4b56-b10b-ee78424e956c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.454321 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.454088 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istiod-ca-cert\") pod \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " Nov 02 21:35:20.454321 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.454183 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istio-token\" (UniqueName: \"kubernetes.io/projected/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istio-token\") pod \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " Nov 02 21:35:20.454321 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.454228 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-workload-socket\") pod \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " Nov 02 21:35:20.454321 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.454258 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istio-data\") pod \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " Nov 02 21:35:20.454737 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.454328 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istio-data\") pod \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " Nov 02 21:35:20.454737 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.454401 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istiod-ca-cert\") pod \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " Nov 02 21:35:20.454737 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.454429 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-credential-socket\") pod \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " Nov 02 21:35:20.454737 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.454459 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istiod-ca-cert" pod "9f07e6bf-04ee-4b56-b10b-ee78424e956c" (UID: "9f07e6bf-04ee-4b56-b10b-ee78424e956c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:20.454737 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.454488 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istio-podinfo\") pod \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " Nov 02 21:35:20.454737 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.454540 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f896b6d0-edd2-4201-a17f-3671ef1517b7-kube-api-access-dscvg\" (UniqueName: \"kubernetes.io/projected/f896b6d0-edd2-4201-a17f-3671ef1517b7-kube-api-access-dscvg\") pod \"f896b6d0-edd2-4201-a17f-3671ef1517b7\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " Nov 02 21:35:20.454737 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.454581 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istio-data" pod "9f07e6bf-04ee-4b56-b10b-ee78424e956c" (UID: "9f07e6bf-04ee-4b56-b10b-ee78424e956c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.454737 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.454606 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-workload-certs\") pod \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\" (UID: \"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33\") " Nov 02 21:35:20.454737 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.454614 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-workload-socket" pod "5ba481e7-6f26-4ae3-91cf-0f2acea6fc33" (UID: "5ba481e7-6f26-4ae3-91cf-0f2acea6fc33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.454737 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.454636 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-istio-data\") pod \"f896b6d0-edd2-4201-a17f-3671ef1517b7\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " Nov 02 21:35:20.454737 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.454664 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-workload-socket\") pod \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " Nov 02 21:35:20.454737 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.454699 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f896b6d0-edd2-4201-a17f-3671ef1517b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f896b6d0-edd2-4201-a17f-3671ef1517b7-istiod-ca-cert\") pod \"f896b6d0-edd2-4201-a17f-3671ef1517b7\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " Nov 02 21:35:20.455371 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.454758 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-workload-certs\") pod \"f896b6d0-edd2-4201-a17f-3671ef1517b7\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " Nov 02 21:35:20.455371 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.454875 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-credential-socket\") pod \"f896b6d0-edd2-4201-a17f-3671ef1517b7\" (UID: \"f896b6d0-edd2-4201-a17f-3671ef1517b7\") " Nov 02 21:35:20.455371 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.454918 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f07e6bf-04ee-4b56-b10b-ee78424e956c-kube-api-access-wczz9\" (UniqueName: \"kubernetes.io/projected/9f07e6bf-04ee-4b56-b10b-ee78424e956c-kube-api-access-wczz9\") pod \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " Nov 02 21:35:20.455371 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.454956 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-workload-certs\") pod \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\" (UID: \"9f07e6bf-04ee-4b56-b10b-ee78424e956c\") " Nov 02 21:35:20.455371 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.454976 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-credential-socket" pod "5ba481e7-6f26-4ae3-91cf-0f2acea6fc33" (UID: "5ba481e7-6f26-4ae3-91cf-0f2acea6fc33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.455608 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.455420 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istiod-ca-cert" pod "5ba481e7-6f26-4ae3-91cf-0f2acea6fc33" (UID: "5ba481e7-6f26-4ae3-91cf-0f2acea6fc33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:20.455608 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.455516 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f896b6d0-edd2-4201-a17f-3671ef1517b7-istiod-ca-cert" pod "f896b6d0-edd2-4201-a17f-3671ef1517b7" (UID: "f896b6d0-edd2-4201-a17f-3671ef1517b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:20.455712 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.455648 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-workload-certs" pod "9f07e6bf-04ee-4b56-b10b-ee78424e956c" (UID: "9f07e6bf-04ee-4b56-b10b-ee78424e956c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.455879 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.455855 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-workload-certs" pod "f896b6d0-edd2-4201-a17f-3671ef1517b7" (UID: "f896b6d0-edd2-4201-a17f-3671ef1517b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.456018 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.455872 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-workload-socket" pod "f896b6d0-edd2-4201-a17f-3671ef1517b7" (UID: "f896b6d0-edd2-4201-a17f-3671ef1517b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.456113 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.456089 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-workload-certs" pod "5ba481e7-6f26-4ae3-91cf-0f2acea6fc33" (UID: "5ba481e7-6f26-4ae3-91cf-0f2acea6fc33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.456265 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.456237 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-credential-socket" pod "f896b6d0-edd2-4201-a17f-3671ef1517b7" (UID: "f896b6d0-edd2-4201-a17f-3671ef1517b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.456382 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.456359 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-istio-data" pod "f896b6d0-edd2-4201-a17f-3671ef1517b7" (UID: "f896b6d0-edd2-4201-a17f-3671ef1517b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.456606 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.456581 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-workload-socket" pod "9f07e6bf-04ee-4b56-b10b-ee78424e956c" (UID: "9f07e6bf-04ee-4b56-b10b-ee78424e956c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.456692 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.456640 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.456692 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.456661 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.456692 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.456675 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.456692 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.456690 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.456897 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.456705 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f896b6d0-edd2-4201-a17f-3671ef1517b7-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.456897 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.456719 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.456897 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.456732 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.456897 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.456746 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.456897 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.456791 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istio-data" pod "5ba481e7-6f26-4ae3-91cf-0f2acea6fc33" (UID: "5ba481e7-6f26-4ae3-91cf-0f2acea6fc33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.458005 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.457975 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f896b6d0-edd2-4201-a17f-3671ef1517b7-kube-api-access-dscvg" pod "f896b6d0-edd2-4201-a17f-3671ef1517b7" (UID: "f896b6d0-edd2-4201-a17f-3671ef1517b7"). InnerVolumeSpecName "kube-api-access-dscvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:20.458195 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.458006 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istio-token" pod "5ba481e7-6f26-4ae3-91cf-0f2acea6fc33" (UID: "5ba481e7-6f26-4ae3-91cf-0f2acea6fc33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:20.458448 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.458415 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istio-envoy" pod "5ba481e7-6f26-4ae3-91cf-0f2acea6fc33" (UID: "5ba481e7-6f26-4ae3-91cf-0f2acea6fc33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.459170 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.459111 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istio-envoy" pod "9f07e6bf-04ee-4b56-b10b-ee78424e956c" (UID: "9f07e6bf-04ee-4b56-b10b-ee78424e956c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.459293 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.459209 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f896b6d0-edd2-4201-a17f-3671ef1517b7-istio-podinfo" pod "f896b6d0-edd2-4201-a17f-3671ef1517b7" (UID: "f896b6d0-edd2-4201-a17f-3671ef1517b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:35:20.459388 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.459358 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istio-token" pod "9f07e6bf-04ee-4b56-b10b-ee78424e956c" (UID: "9f07e6bf-04ee-4b56-b10b-ee78424e956c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:20.459445 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.459395 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istio-podinfo" pod "5ba481e7-6f26-4ae3-91cf-0f2acea6fc33" (UID: "5ba481e7-6f26-4ae3-91cf-0f2acea6fc33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:35:20.459612 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.459579 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-istio-envoy" pod "f896b6d0-edd2-4201-a17f-3671ef1517b7" (UID: "f896b6d0-edd2-4201-a17f-3671ef1517b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:35:20.459818 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.459798 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f896b6d0-edd2-4201-a17f-3671ef1517b7-istio-token" pod "f896b6d0-edd2-4201-a17f-3671ef1517b7" (UID: "f896b6d0-edd2-4201-a17f-3671ef1517b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:20.459893 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.459846 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f07e6bf-04ee-4b56-b10b-ee78424e956c-kube-api-access-wczz9" pod "9f07e6bf-04ee-4b56-b10b-ee78424e956c" (UID: "9f07e6bf-04ee-4b56-b10b-ee78424e956c"). InnerVolumeSpecName "kube-api-access-wczz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:20.460440 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.460420 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-kube-api-access-4cq5b" pod "5ba481e7-6f26-4ae3-91cf-0f2acea6fc33" (UID: "5ba481e7-6f26-4ae3-91cf-0f2acea6fc33"). InnerVolumeSpecName "kube-api-access-4cq5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:20.460734 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.460711 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istio-podinfo" pod "9f07e6bf-04ee-4b56-b10b-ee78424e956c" (UID: "9f07e6bf-04ee-4b56-b10b-ee78424e956c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:35:20.505875 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.505836 4935 generic.go:356] "Generic (PLEG): container finished" podID="5ba481e7-6f26-4ae3-91cf-0f2acea6fc33" containerID="e141428f1d65ff4b072fdffb25e7be307644fdeca98631b39e6193f5685ba17f" exitCode=0 Nov 02 21:35:20.505875 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.505884 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" Nov 02 21:35:20.506239 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.505950 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" event={"ID":"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33","Type":"ContainerDied","Data":"e141428f1d65ff4b072fdffb25e7be307644fdeca98631b39e6193f5685ba17f"} Nov 02 21:35:20.506239 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.505992 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8" event={"ID":"5ba481e7-6f26-4ae3-91cf-0f2acea6fc33","Type":"ContainerDied","Data":"6c1a0d0496db9ceef0040dcb050832ad49fdadb3fa7d18ab7ed1cc06b5d79479"} Nov 02 21:35:20.506239 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.506019 4935 scope.go:122] "RemoveContainer" containerID="e141428f1d65ff4b072fdffb25e7be307644fdeca98631b39e6193f5685ba17f" Nov 02 21:35:20.512007 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.511968 4935 generic.go:356] "Generic (PLEG): container finished" podID="f896b6d0-edd2-4201-a17f-3671ef1517b7" containerID="8530b83c064e4c4140f4592d4935aa62c300378971a82fbef3bb6b2d6c2b1f7c" exitCode=0 Nov 02 21:35:20.512259 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.512021 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" Nov 02 21:35:20.512259 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.512052 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" event={"ID":"f896b6d0-edd2-4201-a17f-3671ef1517b7","Type":"ContainerDied","Data":"8530b83c064e4c4140f4592d4935aa62c300378971a82fbef3bb6b2d6c2b1f7c"} Nov 02 21:35:20.512259 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.512105 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p" event={"ID":"f896b6d0-edd2-4201-a17f-3671ef1517b7","Type":"ContainerDied","Data":"b2fa6d4984f713025af16c56b35a19419daed75dd6f41c90a26e6ae93c4c038d"} Nov 02 21:35:20.517389 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.514027 4935 generic.go:356] "Generic (PLEG): container finished" podID="9f07e6bf-04ee-4b56-b10b-ee78424e956c" containerID="fa6aaabac3bbb08c0b7f65282f3c1be4763bd6eab6035dd092b1010629bf2488" exitCode=0 Nov 02 21:35:20.517389 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.514077 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" Nov 02 21:35:20.517389 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.514102 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" event={"ID":"9f07e6bf-04ee-4b56-b10b-ee78424e956c","Type":"ContainerDied","Data":"fa6aaabac3bbb08c0b7f65282f3c1be4763bd6eab6035dd092b1010629bf2488"} Nov 02 21:35:20.517389 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.514163 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs" event={"ID":"9f07e6bf-04ee-4b56-b10b-ee78424e956c","Type":"ContainerDied","Data":"cdbb512678dd8e517d1fe6823fb314bd72949b7c89b66fcd6650a7f1d3051ead"} Nov 02 21:35:20.524802 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.524719 4935 scope.go:122] "RemoveContainer" containerID="e141428f1d65ff4b072fdffb25e7be307644fdeca98631b39e6193f5685ba17f" Nov 02 21:35:20.525493 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:35:20.525150 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e141428f1d65ff4b072fdffb25e7be307644fdeca98631b39e6193f5685ba17f\": container with ID starting with e141428f1d65ff4b072fdffb25e7be307644fdeca98631b39e6193f5685ba17f not found: ID does not exist" containerID="e141428f1d65ff4b072fdffb25e7be307644fdeca98631b39e6193f5685ba17f" Nov 02 21:35:20.525493 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.525190 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e141428f1d65ff4b072fdffb25e7be307644fdeca98631b39e6193f5685ba17f"} err="failed to get container status \"e141428f1d65ff4b072fdffb25e7be307644fdeca98631b39e6193f5685ba17f\": rpc error: code = NotFound desc = could not find container \"e141428f1d65ff4b072fdffb25e7be307644fdeca98631b39e6193f5685ba17f\": container with ID starting with e141428f1d65ff4b072fdffb25e7be307644fdeca98631b39e6193f5685ba17f not found: ID does not exist" Nov 02 21:35:20.525493 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.525213 4935 scope.go:122] "RemoveContainer" containerID="8530b83c064e4c4140f4592d4935aa62c300378971a82fbef3bb6b2d6c2b1f7c" Nov 02 21:35:20.542475 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.542457 4935 scope.go:122] "RemoveContainer" containerID="8530b83c064e4c4140f4592d4935aa62c300378971a82fbef3bb6b2d6c2b1f7c" Nov 02 21:35:20.542796 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:35:20.542768 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8530b83c064e4c4140f4592d4935aa62c300378971a82fbef3bb6b2d6c2b1f7c\": container with ID starting with 8530b83c064e4c4140f4592d4935aa62c300378971a82fbef3bb6b2d6c2b1f7c not found: ID does not exist" containerID="8530b83c064e4c4140f4592d4935aa62c300378971a82fbef3bb6b2d6c2b1f7c" Nov 02 21:35:20.542857 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.542797 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8530b83c064e4c4140f4592d4935aa62c300378971a82fbef3bb6b2d6c2b1f7c"} err="failed to get container status \"8530b83c064e4c4140f4592d4935aa62c300378971a82fbef3bb6b2d6c2b1f7c\": rpc error: code = NotFound desc = could not find container \"8530b83c064e4c4140f4592d4935aa62c300378971a82fbef3bb6b2d6c2b1f7c\": container with ID starting with 8530b83c064e4c4140f4592d4935aa62c300378971a82fbef3bb6b2d6c2b1f7c not found: ID does not exist" Nov 02 21:35:20.542857 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.542815 4935 scope.go:122] "RemoveContainer" containerID="fa6aaabac3bbb08c0b7f65282f3c1be4763bd6eab6035dd092b1010629bf2488" Nov 02 21:35:20.552638 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.552591 4935 scope.go:122] "RemoveContainer" containerID="fa6aaabac3bbb08c0b7f65282f3c1be4763bd6eab6035dd092b1010629bf2488" Nov 02 21:35:20.552917 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:35:20.552897 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa6aaabac3bbb08c0b7f65282f3c1be4763bd6eab6035dd092b1010629bf2488\": container with ID starting with fa6aaabac3bbb08c0b7f65282f3c1be4763bd6eab6035dd092b1010629bf2488 not found: ID does not exist" containerID="fa6aaabac3bbb08c0b7f65282f3c1be4763bd6eab6035dd092b1010629bf2488" Nov 02 21:35:20.553011 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.552927 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa6aaabac3bbb08c0b7f65282f3c1be4763bd6eab6035dd092b1010629bf2488"} err="failed to get container status \"fa6aaabac3bbb08c0b7f65282f3c1be4763bd6eab6035dd092b1010629bf2488\": rpc error: code = NotFound desc = could not find container \"fa6aaabac3bbb08c0b7f65282f3c1be4763bd6eab6035dd092b1010629bf2488\": container with ID starting with fa6aaabac3bbb08c0b7f65282f3c1be4763bd6eab6035dd092b1010629bf2488 not found: ID does not exist" Nov 02 21:35:20.557927 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.557904 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.558041 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.557931 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dscvg\" (UniqueName: \"kubernetes.io/projected/f896b6d0-edd2-4201-a17f-3671ef1517b7-kube-api-access-dscvg\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.558041 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.557950 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.558041 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.557963 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.558041 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.557976 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.558041 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.557989 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.558041 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.558003 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.558041 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.558017 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wczz9\" (UniqueName: \"kubernetes.io/projected/9f07e6bf-04ee-4b56-b10b-ee78424e956c-kube-api-access-wczz9\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.558041 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.558031 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f896b6d0-edd2-4201-a17f-3671ef1517b7-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.558364 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.558045 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f896b6d0-edd2-4201-a17f-3671ef1517b7-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.558364 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.558059 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.558364 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.558075 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.558364 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.558089 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f07e6bf-04ee-4b56-b10b-ee78424e956c-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.558364 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.558102 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.558364 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.558116 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.558364 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.558147 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f896b6d0-edd2-4201-a17f-3671ef1517b7-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.558364 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.558162 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cq5b\" (UniqueName: \"kubernetes.io/projected/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-kube-api-access-4cq5b\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.558364 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.558176 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.558364 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.558191 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:20.805531 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.805450 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8"] Nov 02 21:35:20.808752 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.808727 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-fjbm8"] Nov 02 21:35:20.910411 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.910375 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p"] Nov 02 21:35:20.916476 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:20.916440 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-kz25p"] Nov 02 21:35:21.019890 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:21.019856 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs"] Nov 02 21:35:21.023269 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:21.023241 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d58d18-openshift-default-866fdc6f49-s76xs"] Nov 02 21:35:21.784334 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:21.784292 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ba481e7-6f26-4ae3-91cf-0f2acea6fc33" path="/var/lib/kubelet/pods/5ba481e7-6f26-4ae3-91cf-0f2acea6fc33/volumes" Nov 02 21:35:21.784963 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:21.784941 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f07e6bf-04ee-4b56-b10b-ee78424e956c" path="/var/lib/kubelet/pods/9f07e6bf-04ee-4b56-b10b-ee78424e956c/volumes" Nov 02 21:35:21.785464 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:21.785444 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f896b6d0-edd2-4201-a17f-3671ef1517b7" path="/var/lib/kubelet/pods/f896b6d0-edd2-4201-a17f-3671ef1517b7/volumes" Nov 02 21:35:22.870160 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:22.870100 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:23.260624 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:23.260522 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:23.471256 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:23.471210 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:24.482746 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:24.482713 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:26.475050 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:26.475007 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:28.471398 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:35:28.471308 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:35:28.836999 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:28.836966 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:29.003441 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:29.003405 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5"] Nov 02 21:35:29.004391 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:29.004332 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5" podUID="b66e9095-274e-449b-b895-8c7b973b2f47" containerName="maas-api" containerID="cri-o://018e154ca3dd432fc10c69d93d7321dcc2c955c79e293646a0000332712e6e9e" gracePeriod=30 Nov 02 21:35:29.155775 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:29.155671 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:29.360163 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:29.360126 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5" Nov 02 21:35:29.486216 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:29.486119 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b66e9095-274e-449b-b895-8c7b973b2f47-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66e9095-274e-449b-b895-8c7b973b2f47-openshift-service-ca\") pod \"b66e9095-274e-449b-b895-8c7b973b2f47\" (UID: \"b66e9095-274e-449b-b895-8c7b973b2f47\") " Nov 02 21:35:29.486216 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:29.486208 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66e9095-274e-449b-b895-8c7b973b2f47-kube-api-access-p29k8\" (UniqueName: \"kubernetes.io/projected/b66e9095-274e-449b-b895-8c7b973b2f47-kube-api-access-p29k8\") pod \"b66e9095-274e-449b-b895-8c7b973b2f47\" (UID: \"b66e9095-274e-449b-b895-8c7b973b2f47\") " Nov 02 21:35:29.486766 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:29.486321 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b66e9095-274e-449b-b895-8c7b973b2f47-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b66e9095-274e-449b-b895-8c7b973b2f47-maas-api-tls\") pod \"b66e9095-274e-449b-b895-8c7b973b2f47\" (UID: \"b66e9095-274e-449b-b895-8c7b973b2f47\") " Nov 02 21:35:29.486766 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:29.486456 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b66e9095-274e-449b-b895-8c7b973b2f47-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b66e9095-274e-449b-b895-8c7b973b2f47-metrics-certs\") pod \"b66e9095-274e-449b-b895-8c7b973b2f47\" (UID: \"b66e9095-274e-449b-b895-8c7b973b2f47\") " Nov 02 21:35:29.486766 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:29.486555 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66e9095-274e-449b-b895-8c7b973b2f47-openshift-service-ca" pod "b66e9095-274e-449b-b895-8c7b973b2f47" (UID: "b66e9095-274e-449b-b895-8c7b973b2f47"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:29.487300 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:29.487281 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66e9095-274e-449b-b895-8c7b973b2f47-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:29.488995 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:29.488959 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b66e9095-274e-449b-b895-8c7b973b2f47-maas-api-tls" pod "b66e9095-274e-449b-b895-8c7b973b2f47" (UID: "b66e9095-274e-449b-b895-8c7b973b2f47"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:29.489112 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:29.489034 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b66e9095-274e-449b-b895-8c7b973b2f47-metrics-certs" pod "b66e9095-274e-449b-b895-8c7b973b2f47" (UID: "b66e9095-274e-449b-b895-8c7b973b2f47"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:29.489226 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:29.489187 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66e9095-274e-449b-b895-8c7b973b2f47-kube-api-access-p29k8" pod "b66e9095-274e-449b-b895-8c7b973b2f47" (UID: "b66e9095-274e-449b-b895-8c7b973b2f47"). InnerVolumeSpecName "kube-api-access-p29k8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:29.588419 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:29.588387 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b66e9095-274e-449b-b895-8c7b973b2f47-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:29.588419 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:29.588416 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b66e9095-274e-449b-b895-8c7b973b2f47-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:29.588419 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:29.588427 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p29k8\" (UniqueName: \"kubernetes.io/projected/b66e9095-274e-449b-b895-8c7b973b2f47-kube-api-access-p29k8\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:29.589976 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:29.589934 4935 generic.go:356] "Generic (PLEG): container finished" podID="b66e9095-274e-449b-b895-8c7b973b2f47" containerID="018e154ca3dd432fc10c69d93d7321dcc2c955c79e293646a0000332712e6e9e" exitCode=0 Nov 02 21:35:29.590170 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:29.589983 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5" Nov 02 21:35:29.590170 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:29.589987 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5" event={"ID":"b66e9095-274e-449b-b895-8c7b973b2f47","Type":"ContainerDied","Data":"018e154ca3dd432fc10c69d93d7321dcc2c955c79e293646a0000332712e6e9e"} Nov 02 21:35:29.590170 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:29.590049 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5" event={"ID":"b66e9095-274e-449b-b895-8c7b973b2f47","Type":"ContainerDied","Data":"229c194f8bb9c5632b7b82ff05603d33c43f1f5459077908ded2470741166e90"} Nov 02 21:35:29.590170 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:29.590075 4935 scope.go:122] "RemoveContainer" containerID="018e154ca3dd432fc10c69d93d7321dcc2c955c79e293646a0000332712e6e9e" Nov 02 21:35:29.607500 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:29.607463 4935 scope.go:122] "RemoveContainer" containerID="018e154ca3dd432fc10c69d93d7321dcc2c955c79e293646a0000332712e6e9e" Nov 02 21:35:29.607874 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:35:29.607847 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"018e154ca3dd432fc10c69d93d7321dcc2c955c79e293646a0000332712e6e9e\": container with ID starting with 018e154ca3dd432fc10c69d93d7321dcc2c955c79e293646a0000332712e6e9e not found: ID does not exist" containerID="018e154ca3dd432fc10c69d93d7321dcc2c955c79e293646a0000332712e6e9e" Nov 02 21:35:29.607964 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:29.607887 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"018e154ca3dd432fc10c69d93d7321dcc2c955c79e293646a0000332712e6e9e"} err="failed to get container status \"018e154ca3dd432fc10c69d93d7321dcc2c955c79e293646a0000332712e6e9e\": rpc error: code = NotFound desc = could not find container \"018e154ca3dd432fc10c69d93d7321dcc2c955c79e293646a0000332712e6e9e\": container with ID starting with 018e154ca3dd432fc10c69d93d7321dcc2c955c79e293646a0000332712e6e9e not found: ID does not exist" Nov 02 21:35:29.744937 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:29.744848 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5"] Nov 02 21:35:29.747934 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:29.747910 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-03ece2-56666dd645-wg9m5"] Nov 02 21:35:29.777787 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:29.777755 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b66e9095-274e-449b-b895-8c7b973b2f47" path="/var/lib/kubelet/pods/b66e9095-274e-449b-b895-8c7b973b2f47/volumes" Nov 02 21:35:30.900000 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:30.899963 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:31.111752 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:31.111715 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:31.337724 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:31.337689 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:32.414741 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:32.414701 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:32.974694 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:32.974654 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:33.631970 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:33.631938 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:33.848354 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:33.848317 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:34.313283 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:34.313247 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:34.908617 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:34.908559 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:35.501449 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:35.501415 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:35.916276 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:35.916239 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:37.608317 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:37.608272 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:37.943899 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:37.943798 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:38.162424 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:38.162390 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:39.624366 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:39.624332 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:39.930598 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:39.930510 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:40.149514 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:40.149473 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:41.203035 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:41.203002 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9"] Nov 02 21:35:41.521432 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:41.521333 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:41.725089 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:41.725052 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:41.942643 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:41.942533 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:43.476847 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:43.476815 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:44.870923 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:44.870870 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:45.072698 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:45.072647 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:45.972546 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:45.972504 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:46.185570 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:46.185535 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:46.209591 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:46.209499 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9" podUID="50e8e193-d60a-4ce2-add8-4e788605bd23" containerName="payload-pre-processing" containerID="cri-o://e44a07b6e078b8cd91d8361b825c551d7b3798ee2c5f3abd3299428fb98b9859" gracePeriod=25 Nov 02 21:35:46.409555 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:46.409517 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:46.513575 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:46.513547 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9" Nov 02 21:35:46.607267 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:46.607233 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50e8e193-d60a-4ce2-add8-4e788605bd23-kube-api-access-qp6j2\" (UniqueName: \"kubernetes.io/projected/50e8e193-d60a-4ce2-add8-4e788605bd23-kube-api-access-qp6j2\") pod \"50e8e193-d60a-4ce2-add8-4e788605bd23\" (UID: \"50e8e193-d60a-4ce2-add8-4e788605bd23\") " Nov 02 21:35:46.607267 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:46.607275 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50e8e193-d60a-4ce2-add8-4e788605bd23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50e8e193-d60a-4ce2-add8-4e788605bd23-plugins-config-volume\") pod \"50e8e193-d60a-4ce2-add8-4e788605bd23\" (UID: \"50e8e193-d60a-4ce2-add8-4e788605bd23\") " Nov 02 21:35:46.607762 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:46.607732 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50e8e193-d60a-4ce2-add8-4e788605bd23-plugins-config-volume" pod "50e8e193-d60a-4ce2-add8-4e788605bd23" (UID: "50e8e193-d60a-4ce2-add8-4e788605bd23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:46.607937 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:46.607920 4935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50e8e193-d60a-4ce2-add8-4e788605bd23-plugins-config-volume\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:46.609965 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:46.609933 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50e8e193-d60a-4ce2-add8-4e788605bd23-kube-api-access-qp6j2" pod "50e8e193-d60a-4ce2-add8-4e788605bd23" (UID: "50e8e193-d60a-4ce2-add8-4e788605bd23"). InnerVolumeSpecName "kube-api-access-qp6j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:46.710403 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:46.710362 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp6j2\" (UniqueName: \"kubernetes.io/projected/50e8e193-d60a-4ce2-add8-4e788605bd23-kube-api-access-qp6j2\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:46.781289 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:46.781251 4935 generic.go:356] "Generic (PLEG): container finished" podID="50e8e193-d60a-4ce2-add8-4e788605bd23" containerID="e44a07b6e078b8cd91d8361b825c551d7b3798ee2c5f3abd3299428fb98b9859" exitCode=0 Nov 02 21:35:46.781459 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:46.781294 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9" Nov 02 21:35:46.781459 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:46.781336 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9" event={"ID":"50e8e193-d60a-4ce2-add8-4e788605bd23","Type":"ContainerDied","Data":"e44a07b6e078b8cd91d8361b825c551d7b3798ee2c5f3abd3299428fb98b9859"} Nov 02 21:35:46.781459 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:46.781388 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9" event={"ID":"50e8e193-d60a-4ce2-add8-4e788605bd23","Type":"ContainerDied","Data":"1a09852ec154735e6bb3ff6567d7b4261b731c81957c0e2e2e8332f2239c0108"} Nov 02 21:35:46.781459 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:46.781410 4935 scope.go:122] "RemoveContainer" containerID="e44a07b6e078b8cd91d8361b825c551d7b3798ee2c5f3abd3299428fb98b9859" Nov 02 21:35:46.792360 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:46.792339 4935 scope.go:122] "RemoveContainer" containerID="e44a07b6e078b8cd91d8361b825c551d7b3798ee2c5f3abd3299428fb98b9859" Nov 02 21:35:46.792651 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:35:46.792626 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e44a07b6e078b8cd91d8361b825c551d7b3798ee2c5f3abd3299428fb98b9859\": container with ID starting with e44a07b6e078b8cd91d8361b825c551d7b3798ee2c5f3abd3299428fb98b9859 not found: ID does not exist" containerID="e44a07b6e078b8cd91d8361b825c551d7b3798ee2c5f3abd3299428fb98b9859" Nov 02 21:35:46.792711 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:46.792660 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e44a07b6e078b8cd91d8361b825c551d7b3798ee2c5f3abd3299428fb98b9859"} err="failed to get container status \"e44a07b6e078b8cd91d8361b825c551d7b3798ee2c5f3abd3299428fb98b9859\": rpc error: code = NotFound desc = could not find container \"e44a07b6e078b8cd91d8361b825c551d7b3798ee2c5f3abd3299428fb98b9859\": container with ID starting with e44a07b6e078b8cd91d8361b825c551d7b3798ee2c5f3abd3299428fb98b9859 not found: ID does not exist" Nov 02 21:35:46.940231 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:46.940190 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9"] Nov 02 21:35:46.943974 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:46.943944 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bd96ef-5bffb4d8f5-7kcp9"] Nov 02 21:35:47.490887 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:47.490841 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:47.779856 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:47.779818 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50e8e193-d60a-4ce2-add8-4e788605bd23" path="/var/lib/kubelet/pods/50e8e193-d60a-4ce2-add8-4e788605bd23/volumes" Nov 02 21:35:49.072738 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:49.072698 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:49.620936 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:49.620900 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:49.834592 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:49.834519 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:54.211794 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:54.211757 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:57.504935 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:57.504870 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:35:58.470519 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:35:58.470477 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:35:59.328758 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:35:59.328722 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:04.432035 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:04.431999 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:08.649095 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:08.649014 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g"] Nov 02 21:36:08.649596 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:08.649233 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g" podUID="d34e34f3-77b3-433c-8e39-07d64e93fb32" containerName="maas-api" containerID="cri-o://3f978fba62849a05db0bb0e6f515195e092d4dfc5fe6461201899ae43818d806" gracePeriod=30 Nov 02 21:36:08.832798 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:08.832758 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:08.964504 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:08.964425 4935 generic.go:356] "Generic (PLEG): container finished" podID="d34e34f3-77b3-433c-8e39-07d64e93fb32" containerID="3f978fba62849a05db0bb0e6f515195e092d4dfc5fe6461201899ae43818d806" exitCode=0 Nov 02 21:36:08.964759 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:08.964511 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g" event={"ID":"d34e34f3-77b3-433c-8e39-07d64e93fb32","Type":"ContainerDied","Data":"3f978fba62849a05db0bb0e6f515195e092d4dfc5fe6461201899ae43818d806"} Nov 02 21:36:08.993611 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:08.993583 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g" Nov 02 21:36:09.044302 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:09.042816 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d34e34f3-77b3-433c-8e39-07d64e93fb32-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d34e34f3-77b3-433c-8e39-07d64e93fb32-metrics-certs\") pod \"d34e34f3-77b3-433c-8e39-07d64e93fb32\" (UID: \"d34e34f3-77b3-433c-8e39-07d64e93fb32\") " Nov 02 21:36:09.044302 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:09.042981 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d34e34f3-77b3-433c-8e39-07d64e93fb32-kube-api-access-85mrf\" (UniqueName: \"kubernetes.io/projected/d34e34f3-77b3-433c-8e39-07d64e93fb32-kube-api-access-85mrf\") pod \"d34e34f3-77b3-433c-8e39-07d64e93fb32\" (UID: \"d34e34f3-77b3-433c-8e39-07d64e93fb32\") " Nov 02 21:36:09.044302 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:09.043051 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d34e34f3-77b3-433c-8e39-07d64e93fb32-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d34e34f3-77b3-433c-8e39-07d64e93fb32-openshift-service-ca\") pod \"d34e34f3-77b3-433c-8e39-07d64e93fb32\" (UID: \"d34e34f3-77b3-433c-8e39-07d64e93fb32\") " Nov 02 21:36:09.044302 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:09.043099 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d34e34f3-77b3-433c-8e39-07d64e93fb32-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d34e34f3-77b3-433c-8e39-07d64e93fb32-maas-api-tls\") pod \"d34e34f3-77b3-433c-8e39-07d64e93fb32\" (UID: \"d34e34f3-77b3-433c-8e39-07d64e93fb32\") " Nov 02 21:36:09.047019 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:09.045385 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d34e34f3-77b3-433c-8e39-07d64e93fb32-openshift-service-ca" pod "d34e34f3-77b3-433c-8e39-07d64e93fb32" (UID: "d34e34f3-77b3-433c-8e39-07d64e93fb32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:36:09.049072 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:09.048287 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d34e34f3-77b3-433c-8e39-07d64e93fb32-maas-api-tls" pod "d34e34f3-77b3-433c-8e39-07d64e93fb32" (UID: "d34e34f3-77b3-433c-8e39-07d64e93fb32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:36:09.052641 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:09.052615 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d34e34f3-77b3-433c-8e39-07d64e93fb32-metrics-certs" pod "d34e34f3-77b3-433c-8e39-07d64e93fb32" (UID: "d34e34f3-77b3-433c-8e39-07d64e93fb32"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:36:09.061396 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:09.061368 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d34e34f3-77b3-433c-8e39-07d64e93fb32-kube-api-access-85mrf" pod "d34e34f3-77b3-433c-8e39-07d64e93fb32" (UID: "d34e34f3-77b3-433c-8e39-07d64e93fb32"). InnerVolumeSpecName "kube-api-access-85mrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:36:09.146328 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:09.146285 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85mrf\" (UniqueName: \"kubernetes.io/projected/d34e34f3-77b3-433c-8e39-07d64e93fb32-kube-api-access-85mrf\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:09.146328 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:09.146329 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d34e34f3-77b3-433c-8e39-07d64e93fb32-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:09.146537 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:09.146347 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d34e34f3-77b3-433c-8e39-07d64e93fb32-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:09.146537 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:09.146361 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d34e34f3-77b3-433c-8e39-07d64e93fb32-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:09.980575 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:09.980543 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g" Nov 02 21:36:09.980575 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:09.980550 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g" event={"ID":"d34e34f3-77b3-433c-8e39-07d64e93fb32","Type":"ContainerDied","Data":"11cb25979473d1dfb1aacaef57c632da77619879590d806611f89b0b7aff8fbc"} Nov 02 21:36:09.981092 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:09.980639 4935 scope.go:122] "RemoveContainer" containerID="3f978fba62849a05db0bb0e6f515195e092d4dfc5fe6461201899ae43818d806" Nov 02 21:36:10.134349 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:10.134272 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g"] Nov 02 21:36:10.180295 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:10.180264 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-54a4b2-574cd679f6-fgb9g"] Nov 02 21:36:10.657855 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:10.657823 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:10.872849 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:10.872807 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:11.777964 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:11.777917 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d34e34f3-77b3-433c-8e39-07d64e93fb32" path="/var/lib/kubelet/pods/d34e34f3-77b3-433c-8e39-07d64e93fb32/volumes" Nov 02 21:36:12.237884 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:12.237802 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:12.751537 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:12.751495 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:16.126595 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:16.126556 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:16.937286 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:16.937248 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:17.148395 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:17.148347 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:18.805159 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:18.805108 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:19.461167 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:19.459036 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:19.906723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:19.906683 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:21.553613 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:21.553579 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:21.770323 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:21.770278 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:21.984450 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:21.984362 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:23.442027 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:23.441990 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:23.717205 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:23.715087 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:26.253904 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:26.253864 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79"] Nov 02 21:36:26.254492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:26.254154 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" podUID="5df21a5e-97fc-4564-8118-884d34b0d3d1" containerName="istio-proxy" containerID="cri-o://eb99a3abce76100f7cf0fd5b6de44d27604ebd23aa951b3e32f952ce96124f7f" gracePeriod=30 Nov 02 21:36:26.298551 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:26.298506 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k"] Nov 02 21:36:26.298778 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:26.298732 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" podUID="7f86cdaf-d4dd-4281-ad28-f2750dab88a9" containerName="istio-proxy" containerID="cri-o://ee6b4830c628e6e019b85b522551dbda2238ce2a6a6cb461050d44df07f4a0be" gracePeriod=30 Nov 02 21:36:26.630241 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:26.630091 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:26.869545 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:26.866330 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:27.436921 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:27.436865 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:27.656184 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:27.656116 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:27.860004 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:27.859962 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:28.183642 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:28.183565 4935 scope.go:122] "RemoveContainer" containerID="0a0879756b1082f42a72a10d26f321fa19942903a230fe152906f2d386058fc0" Nov 02 21:36:28.470646 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:36:28.470549 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:36:29.598242 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:29.598204 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6"] Nov 02 21:36:29.598937 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:29.598893 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" podUID="49d7528a-f6fa-49f0-97a7-461066310602" containerName="istio-proxy" containerID="cri-o://94e12b06d4c1e7e110bbdaf8d16b44cc115fdc4e3f4c98b007f1254bd90e3900" gracePeriod=30 Nov 02 21:36:29.644033 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:29.643999 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn"] Nov 02 21:36:29.644248 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:29.644224 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" podUID="83fbff2a-a71b-40bd-a636-58c762e22aee" containerName="istio-proxy" containerID="cri-o://361105a3c6d5b601fdc89e15c2faee4a4a856032b88b3452ce8c32b8d0b6982f" gracePeriod=30 Nov 02 21:36:31.342153 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.336730 4935 patch_prober.go:29] interesting pod/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 02 21:36:31.342153 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.336804 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" podUID="5df21a5e-97fc-4564-8118-884d34b0d3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 02 21:36:31.623827 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.623794 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:36:31.649912 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.649886 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:36:31.693009 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.692980 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-workload-socket\") pod \"5df21a5e-97fc-4564-8118-884d34b0d3d1\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " Nov 02 21:36:31.693232 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.693026 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-istio-data\") pod \"5df21a5e-97fc-4564-8118-884d34b0d3d1\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " Nov 02 21:36:31.693232 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.693153 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5df21a5e-97fc-4564-8118-884d34b0d3d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5df21a5e-97fc-4564-8118-884d34b0d3d1-istiod-ca-cert\") pod \"5df21a5e-97fc-4564-8118-884d34b0d3d1\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " Nov 02 21:36:31.693232 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.693189 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5df21a5e-97fc-4564-8118-884d34b0d3d1-kube-api-access-j8m6d\" (UniqueName: \"kubernetes.io/projected/5df21a5e-97fc-4564-8118-884d34b0d3d1-kube-api-access-j8m6d\") pod \"5df21a5e-97fc-4564-8118-884d34b0d3d1\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " Nov 02 21:36:31.693397 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.693319 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-workload-socket" pod "5df21a5e-97fc-4564-8118-884d34b0d3d1" (UID: "5df21a5e-97fc-4564-8118-884d34b0d3d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:36:31.693397 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.693349 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5df21a5e-97fc-4564-8118-884d34b0d3d1-istio-token\" (UniqueName: \"kubernetes.io/projected/5df21a5e-97fc-4564-8118-884d34b0d3d1-istio-token\") pod \"5df21a5e-97fc-4564-8118-884d34b0d3d1\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " Nov 02 21:36:31.693397 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.693375 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-istio-data" pod "5df21a5e-97fc-4564-8118-884d34b0d3d1" (UID: "5df21a5e-97fc-4564-8118-884d34b0d3d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:36:31.693561 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.693397 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-credential-socket\") pod \"5df21a5e-97fc-4564-8118-884d34b0d3d1\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " Nov 02 21:36:31.693561 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.693465 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-workload-certs\") pod \"5df21a5e-97fc-4564-8118-884d34b0d3d1\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " Nov 02 21:36:31.693561 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.693508 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-istio-envoy\") pod \"5df21a5e-97fc-4564-8118-884d34b0d3d1\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " Nov 02 21:36:31.693561 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.693542 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5df21a5e-97fc-4564-8118-884d34b0d3d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5df21a5e-97fc-4564-8118-884d34b0d3d1-istio-podinfo\") pod \"5df21a5e-97fc-4564-8118-884d34b0d3d1\" (UID: \"5df21a5e-97fc-4564-8118-884d34b0d3d1\") " Nov 02 21:36:31.693766 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.693603 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5df21a5e-97fc-4564-8118-884d34b0d3d1-istiod-ca-cert" pod "5df21a5e-97fc-4564-8118-884d34b0d3d1" (UID: "5df21a5e-97fc-4564-8118-884d34b0d3d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:36:31.693766 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.693666 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-credential-socket" pod "5df21a5e-97fc-4564-8118-884d34b0d3d1" (UID: "5df21a5e-97fc-4564-8118-884d34b0d3d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:36:31.693766 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.693750 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-workload-certs" pod "5df21a5e-97fc-4564-8118-884d34b0d3d1" (UID: "5df21a5e-97fc-4564-8118-884d34b0d3d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:36:31.694212 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.694187 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:31.694348 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.694216 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:31.694348 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.694232 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:31.694348 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.694246 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:31.694348 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.694261 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5df21a5e-97fc-4564-8118-884d34b0d3d1-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:31.695804 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.695782 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-istio-envoy" pod "5df21a5e-97fc-4564-8118-884d34b0d3d1" (UID: "5df21a5e-97fc-4564-8118-884d34b0d3d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:36:31.695906 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.695837 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df21a5e-97fc-4564-8118-884d34b0d3d1-kube-api-access-j8m6d" pod "5df21a5e-97fc-4564-8118-884d34b0d3d1" (UID: "5df21a5e-97fc-4564-8118-884d34b0d3d1"). InnerVolumeSpecName "kube-api-access-j8m6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:36:31.695951 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.695924 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5df21a5e-97fc-4564-8118-884d34b0d3d1-istio-podinfo" pod "5df21a5e-97fc-4564-8118-884d34b0d3d1" (UID: "5df21a5e-97fc-4564-8118-884d34b0d3d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:36:31.695951 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.695934 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df21a5e-97fc-4564-8118-884d34b0d3d1-istio-token" pod "5df21a5e-97fc-4564-8118-884d34b0d3d1" (UID: "5df21a5e-97fc-4564-8118-884d34b0d3d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:36:31.796075 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.795871 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-credential-socket\") pod \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " Nov 02 21:36:31.796075 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.795934 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-workload-certs\") pod \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " Nov 02 21:36:31.796075 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.795998 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istiod-ca-cert\") pod \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " Nov 02 21:36:31.796075 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.796026 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istio-token\" (UniqueName: \"kubernetes.io/projected/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istio-token\") pod \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " Nov 02 21:36:31.796434 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.796165 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istio-data\") pod \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " Nov 02 21:36:31.796434 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.796232 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-credential-socket" pod "7f86cdaf-d4dd-4281-ad28-f2750dab88a9" (UID: "7f86cdaf-d4dd-4281-ad28-f2750dab88a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:36:31.796434 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.796242 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-kube-api-access-7gqwq\" (UniqueName: \"kubernetes.io/projected/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-kube-api-access-7gqwq\") pod \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " Nov 02 21:36:31.799006 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.796717 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-workload-certs" pod "7f86cdaf-d4dd-4281-ad28-f2750dab88a9" (UID: "7f86cdaf-d4dd-4281-ad28-f2750dab88a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:36:31.799006 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.796967 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istio-data" pod "7f86cdaf-d4dd-4281-ad28-f2750dab88a9" (UID: "7f86cdaf-d4dd-4281-ad28-f2750dab88a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:36:31.799006 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.797231 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istio-podinfo\") pod \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " Nov 02 21:36:31.799006 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.797257 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istiod-ca-cert" pod "7f86cdaf-d4dd-4281-ad28-f2750dab88a9" (UID: "7f86cdaf-d4dd-4281-ad28-f2750dab88a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:36:31.799006 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.797277 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istio-envoy\") pod \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " Nov 02 21:36:31.799006 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.797313 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-workload-socket\") pod \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\" (UID: \"7f86cdaf-d4dd-4281-ad28-f2750dab88a9\") " Nov 02 21:36:31.799006 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.797866 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:31.799006 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.797890 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:31.799006 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.797905 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8m6d\" (UniqueName: \"kubernetes.io/projected/5df21a5e-97fc-4564-8118-884d34b0d3d1-kube-api-access-j8m6d\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:31.799006 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.797920 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:31.799006 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.797933 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:31.799006 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.797948 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5df21a5e-97fc-4564-8118-884d34b0d3d1-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:31.799006 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.797961 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5df21a5e-97fc-4564-8118-884d34b0d3d1-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:31.799006 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.797975 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5df21a5e-97fc-4564-8118-884d34b0d3d1-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:31.799006 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.798264 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-workload-socket" pod "7f86cdaf-d4dd-4281-ad28-f2750dab88a9" (UID: "7f86cdaf-d4dd-4281-ad28-f2750dab88a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:36:31.801036 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.801006 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istio-podinfo" pod "7f86cdaf-d4dd-4281-ad28-f2750dab88a9" (UID: "7f86cdaf-d4dd-4281-ad28-f2750dab88a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:36:31.801612 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.801584 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istio-token" pod "7f86cdaf-d4dd-4281-ad28-f2750dab88a9" (UID: "7f86cdaf-d4dd-4281-ad28-f2750dab88a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:36:31.802496 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.802470 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-kube-api-access-7gqwq" pod "7f86cdaf-d4dd-4281-ad28-f2750dab88a9" (UID: "7f86cdaf-d4dd-4281-ad28-f2750dab88a9"). InnerVolumeSpecName "kube-api-access-7gqwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:36:31.802680 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.802663 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istio-envoy" pod "7f86cdaf-d4dd-4281-ad28-f2750dab88a9" (UID: "7f86cdaf-d4dd-4281-ad28-f2750dab88a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:36:31.905629 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.902956 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:31.905629 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.902996 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gqwq\" (UniqueName: \"kubernetes.io/projected/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-kube-api-access-7gqwq\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:31.905629 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.903012 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:31.905629 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.903038 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:31.905629 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:31.903053 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f86cdaf-d4dd-4281-ad28-f2750dab88a9-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:32.180988 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:32.180887 4935 generic.go:356] "Generic (PLEG): container finished" podID="5df21a5e-97fc-4564-8118-884d34b0d3d1" containerID="eb99a3abce76100f7cf0fd5b6de44d27604ebd23aa951b3e32f952ce96124f7f" exitCode=0 Nov 02 21:36:32.180988 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:32.180969 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" Nov 02 21:36:32.181236 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:32.180975 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" event={"ID":"5df21a5e-97fc-4564-8118-884d34b0d3d1","Type":"ContainerDied","Data":"eb99a3abce76100f7cf0fd5b6de44d27604ebd23aa951b3e32f952ce96124f7f"} Nov 02 21:36:32.181236 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:32.181043 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79" event={"ID":"5df21a5e-97fc-4564-8118-884d34b0d3d1","Type":"ContainerDied","Data":"11ab4fef019663f83bc172198a3791de7d2aa2aa7a06a4aedbe12f0c36d4454e"} Nov 02 21:36:32.181236 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:32.181074 4935 scope.go:122] "RemoveContainer" containerID="eb99a3abce76100f7cf0fd5b6de44d27604ebd23aa951b3e32f952ce96124f7f" Nov 02 21:36:32.182673 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:32.182646 4935 generic.go:356] "Generic (PLEG): container finished" podID="7f86cdaf-d4dd-4281-ad28-f2750dab88a9" containerID="ee6b4830c628e6e019b85b522551dbda2238ce2a6a6cb461050d44df07f4a0be" exitCode=0 Nov 02 21:36:32.182847 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:32.182720 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" event={"ID":"7f86cdaf-d4dd-4281-ad28-f2750dab88a9","Type":"ContainerDied","Data":"ee6b4830c628e6e019b85b522551dbda2238ce2a6a6cb461050d44df07f4a0be"} Nov 02 21:36:32.182847 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:32.182772 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" event={"ID":"7f86cdaf-d4dd-4281-ad28-f2750dab88a9","Type":"ContainerDied","Data":"6d9b96d475bef3096f7b22961b24c261d030ef5059b8abbb413bd69304c99372"} Nov 02 21:36:32.182847 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:32.182788 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k" Nov 02 21:36:32.196497 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:32.196476 4935 scope.go:122] "RemoveContainer" containerID="eb99a3abce76100f7cf0fd5b6de44d27604ebd23aa951b3e32f952ce96124f7f" Nov 02 21:36:32.196826 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:36:32.196801 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb99a3abce76100f7cf0fd5b6de44d27604ebd23aa951b3e32f952ce96124f7f\": container with ID starting with eb99a3abce76100f7cf0fd5b6de44d27604ebd23aa951b3e32f952ce96124f7f not found: ID does not exist" containerID="eb99a3abce76100f7cf0fd5b6de44d27604ebd23aa951b3e32f952ce96124f7f" Nov 02 21:36:32.196906 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:32.196832 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb99a3abce76100f7cf0fd5b6de44d27604ebd23aa951b3e32f952ce96124f7f"} err="failed to get container status \"eb99a3abce76100f7cf0fd5b6de44d27604ebd23aa951b3e32f952ce96124f7f\": rpc error: code = NotFound desc = could not find container \"eb99a3abce76100f7cf0fd5b6de44d27604ebd23aa951b3e32f952ce96124f7f\": container with ID starting with eb99a3abce76100f7cf0fd5b6de44d27604ebd23aa951b3e32f952ce96124f7f not found: ID does not exist" Nov 02 21:36:32.196906 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:32.196850 4935 scope.go:122] "RemoveContainer" containerID="ee6b4830c628e6e019b85b522551dbda2238ce2a6a6cb461050d44df07f4a0be" Nov 02 21:36:32.206962 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:32.206941 4935 scope.go:122] "RemoveContainer" containerID="ee6b4830c628e6e019b85b522551dbda2238ce2a6a6cb461050d44df07f4a0be" Nov 02 21:36:32.207328 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:36:32.207302 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee6b4830c628e6e019b85b522551dbda2238ce2a6a6cb461050d44df07f4a0be\": container with ID starting with ee6b4830c628e6e019b85b522551dbda2238ce2a6a6cb461050d44df07f4a0be not found: ID does not exist" containerID="ee6b4830c628e6e019b85b522551dbda2238ce2a6a6cb461050d44df07f4a0be" Nov 02 21:36:32.207410 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:32.207334 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee6b4830c628e6e019b85b522551dbda2238ce2a6a6cb461050d44df07f4a0be"} err="failed to get container status \"ee6b4830c628e6e019b85b522551dbda2238ce2a6a6cb461050d44df07f4a0be\": rpc error: code = NotFound desc = could not find container \"ee6b4830c628e6e019b85b522551dbda2238ce2a6a6cb461050d44df07f4a0be\": container with ID starting with ee6b4830c628e6e019b85b522551dbda2238ce2a6a6cb461050d44df07f4a0be not found: ID does not exist" Nov 02 21:36:32.271093 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:32.271060 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:32.347180 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:32.337928 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k"] Nov 02 21:36:32.347180 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:32.343350 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-jhd7k"] Nov 02 21:36:32.444184 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:32.444079 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79"] Nov 02 21:36:32.450185 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:32.450152 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-pmh79"] Nov 02 21:36:33.777147 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:33.777109 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5df21a5e-97fc-4564-8118-884d34b0d3d1" path="/var/lib/kubelet/pods/5df21a5e-97fc-4564-8118-884d34b0d3d1/volumes" Nov 02 21:36:33.777587 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:33.777570 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f86cdaf-d4dd-4281-ad28-f2750dab88a9" path="/var/lib/kubelet/pods/7f86cdaf-d4dd-4281-ad28-f2750dab88a9/volumes" Nov 02 21:36:34.108948 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:34.108861 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:34.933167 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:34.933122 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:36:34.971963 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:34.971935 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:36:35.044295 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.044267 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-istio-envoy\") pod \"83fbff2a-a71b-40bd-a636-58c762e22aee\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " Nov 02 21:36:35.044295 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.044299 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-credential-socket\") pod \"83fbff2a-a71b-40bd-a636-58c762e22aee\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " Nov 02 21:36:35.044529 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.044320 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-istio-data\") pod \"83fbff2a-a71b-40bd-a636-58c762e22aee\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " Nov 02 21:36:35.044529 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.044373 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49d7528a-f6fa-49f0-97a7-461066310602-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49d7528a-f6fa-49f0-97a7-461066310602-istio-podinfo\") pod \"49d7528a-f6fa-49f0-97a7-461066310602\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " Nov 02 21:36:35.044529 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.044400 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83fbff2a-a71b-40bd-a636-58c762e22aee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83fbff2a-a71b-40bd-a636-58c762e22aee-istiod-ca-cert\") pod \"83fbff2a-a71b-40bd-a636-58c762e22aee\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " Nov 02 21:36:35.044529 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.044436 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-workload-certs\") pod \"49d7528a-f6fa-49f0-97a7-461066310602\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " Nov 02 21:36:35.044529 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.044468 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-workload-certs\") pod \"83fbff2a-a71b-40bd-a636-58c762e22aee\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " Nov 02 21:36:35.044780 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.044593 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83fbff2a-a71b-40bd-a636-58c762e22aee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83fbff2a-a71b-40bd-a636-58c762e22aee-istio-podinfo\") pod \"83fbff2a-a71b-40bd-a636-58c762e22aee\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " Nov 02 21:36:35.044780 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.044642 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-credential-socket\") pod \"49d7528a-f6fa-49f0-97a7-461066310602\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " Nov 02 21:36:35.044780 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.044616 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-credential-socket" pod "83fbff2a-a71b-40bd-a636-58c762e22aee" (UID: "83fbff2a-a71b-40bd-a636-58c762e22aee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:36:35.044780 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.044671 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-workload-socket\") pod \"49d7528a-f6fa-49f0-97a7-461066310602\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " Nov 02 21:36:35.044780 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.044705 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-workload-socket\") pod \"83fbff2a-a71b-40bd-a636-58c762e22aee\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " Nov 02 21:36:35.044780 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.044688 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-istio-data" pod "83fbff2a-a71b-40bd-a636-58c762e22aee" (UID: "83fbff2a-a71b-40bd-a636-58c762e22aee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:36:35.044780 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.044740 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49d7528a-f6fa-49f0-97a7-461066310602-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49d7528a-f6fa-49f0-97a7-461066310602-istiod-ca-cert\") pod \"49d7528a-f6fa-49f0-97a7-461066310602\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " Nov 02 21:36:35.044780 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.044758 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-workload-certs" pod "49d7528a-f6fa-49f0-97a7-461066310602" (UID: "49d7528a-f6fa-49f0-97a7-461066310602"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:36:35.044780 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.044773 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-workload-certs" pod "83fbff2a-a71b-40bd-a636-58c762e22aee" (UID: "83fbff2a-a71b-40bd-a636-58c762e22aee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:36:35.045264 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.044789 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-istio-envoy\") pod \"49d7528a-f6fa-49f0-97a7-461066310602\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " Nov 02 21:36:35.045264 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.044792 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83fbff2a-a71b-40bd-a636-58c762e22aee-istiod-ca-cert" pod "83fbff2a-a71b-40bd-a636-58c762e22aee" (UID: "83fbff2a-a71b-40bd-a636-58c762e22aee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:36:35.045264 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.044833 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-istio-data\") pod \"49d7528a-f6fa-49f0-97a7-461066310602\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " Nov 02 21:36:35.045264 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.044874 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49d7528a-f6fa-49f0-97a7-461066310602-kube-api-access-7hdkw\" (UniqueName: \"kubernetes.io/projected/49d7528a-f6fa-49f0-97a7-461066310602-kube-api-access-7hdkw\") pod \"49d7528a-f6fa-49f0-97a7-461066310602\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " Nov 02 21:36:35.045264 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.044905 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-credential-socket" pod "49d7528a-f6fa-49f0-97a7-461066310602" (UID: "49d7528a-f6fa-49f0-97a7-461066310602"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:36:35.045264 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.044940 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83fbff2a-a71b-40bd-a636-58c762e22aee-istio-token\" (UniqueName: \"kubernetes.io/projected/83fbff2a-a71b-40bd-a636-58c762e22aee-istio-token\") pod \"83fbff2a-a71b-40bd-a636-58c762e22aee\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " Nov 02 21:36:35.045264 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.044954 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-workload-socket" pod "83fbff2a-a71b-40bd-a636-58c762e22aee" (UID: "83fbff2a-a71b-40bd-a636-58c762e22aee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:36:35.045264 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.044988 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49d7528a-f6fa-49f0-97a7-461066310602-istio-token\" (UniqueName: \"kubernetes.io/projected/49d7528a-f6fa-49f0-97a7-461066310602-istio-token\") pod \"49d7528a-f6fa-49f0-97a7-461066310602\" (UID: \"49d7528a-f6fa-49f0-97a7-461066310602\") " Nov 02 21:36:35.045264 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.045031 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-workload-socket" pod "49d7528a-f6fa-49f0-97a7-461066310602" (UID: "49d7528a-f6fa-49f0-97a7-461066310602"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:36:35.045264 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.045044 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83fbff2a-a71b-40bd-a636-58c762e22aee-kube-api-access-9qwfg\" (UniqueName: \"kubernetes.io/projected/83fbff2a-a71b-40bd-a636-58c762e22aee-kube-api-access-9qwfg\") pod \"83fbff2a-a71b-40bd-a636-58c762e22aee\" (UID: \"83fbff2a-a71b-40bd-a636-58c762e22aee\") " Nov 02 21:36:35.045264 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.045213 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-istio-data" pod "49d7528a-f6fa-49f0-97a7-461066310602" (UID: "49d7528a-f6fa-49f0-97a7-461066310602"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:36:35.045807 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.045261 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49d7528a-f6fa-49f0-97a7-461066310602-istiod-ca-cert" pod "49d7528a-f6fa-49f0-97a7-461066310602" (UID: "49d7528a-f6fa-49f0-97a7-461066310602"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:36:35.045807 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.045701 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49d7528a-f6fa-49f0-97a7-461066310602-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:35.045807 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.045724 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:35.045807 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.045741 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:35.045807 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.045756 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:35.045807 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.045773 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83fbff2a-a71b-40bd-a636-58c762e22aee-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:35.045807 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.045789 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:35.045807 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.045803 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:35.046215 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.045818 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:35.046215 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.045831 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:35.046215 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.045846 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:35.047040 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.046991 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49d7528a-f6fa-49f0-97a7-461066310602-istio-podinfo" pod "49d7528a-f6fa-49f0-97a7-461066310602" (UID: "49d7528a-f6fa-49f0-97a7-461066310602"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:36:35.047504 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.047484 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83fbff2a-a71b-40bd-a636-58c762e22aee-istio-podinfo" pod "83fbff2a-a71b-40bd-a636-58c762e22aee" (UID: "83fbff2a-a71b-40bd-a636-58c762e22aee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:36:35.047597 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.047523 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d7528a-f6fa-49f0-97a7-461066310602-istio-token" pod "49d7528a-f6fa-49f0-97a7-461066310602" (UID: "49d7528a-f6fa-49f0-97a7-461066310602"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:36:35.047758 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.047725 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-istio-envoy" pod "49d7528a-f6fa-49f0-97a7-461066310602" (UID: "49d7528a-f6fa-49f0-97a7-461066310602"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:36:35.048544 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.048519 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d7528a-f6fa-49f0-97a7-461066310602-kube-api-access-7hdkw" pod "49d7528a-f6fa-49f0-97a7-461066310602" (UID: "49d7528a-f6fa-49f0-97a7-461066310602"). InnerVolumeSpecName "kube-api-access-7hdkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:36:35.048630 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.048565 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-istio-envoy" pod "83fbff2a-a71b-40bd-a636-58c762e22aee" (UID: "83fbff2a-a71b-40bd-a636-58c762e22aee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:36:35.048630 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.048573 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83fbff2a-a71b-40bd-a636-58c762e22aee-istio-token" pod "83fbff2a-a71b-40bd-a636-58c762e22aee" (UID: "83fbff2a-a71b-40bd-a636-58c762e22aee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:36:35.049058 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.049037 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83fbff2a-a71b-40bd-a636-58c762e22aee-kube-api-access-9qwfg" pod "83fbff2a-a71b-40bd-a636-58c762e22aee" (UID: "83fbff2a-a71b-40bd-a636-58c762e22aee"). InnerVolumeSpecName "kube-api-access-9qwfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:36:35.147436 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.147351 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83fbff2a-a71b-40bd-a636-58c762e22aee-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:35.147436 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.147379 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49d7528a-f6fa-49f0-97a7-461066310602-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:35.147436 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.147389 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hdkw\" (UniqueName: \"kubernetes.io/projected/49d7528a-f6fa-49f0-97a7-461066310602-kube-api-access-7hdkw\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:35.147436 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.147398 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83fbff2a-a71b-40bd-a636-58c762e22aee-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:35.147436 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.147407 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49d7528a-f6fa-49f0-97a7-461066310602-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:35.147436 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.147418 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qwfg\" (UniqueName: \"kubernetes.io/projected/83fbff2a-a71b-40bd-a636-58c762e22aee-kube-api-access-9qwfg\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:35.147436 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.147429 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83fbff2a-a71b-40bd-a636-58c762e22aee-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:35.147436 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.147437 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49d7528a-f6fa-49f0-97a7-461066310602-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:35.212369 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.212326 4935 generic.go:356] "Generic (PLEG): container finished" podID="49d7528a-f6fa-49f0-97a7-461066310602" containerID="94e12b06d4c1e7e110bbdaf8d16b44cc115fdc4e3f4c98b007f1254bd90e3900" exitCode=0 Nov 02 21:36:35.212369 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.212369 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" Nov 02 21:36:35.212618 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.212385 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" event={"ID":"49d7528a-f6fa-49f0-97a7-461066310602","Type":"ContainerDied","Data":"94e12b06d4c1e7e110bbdaf8d16b44cc115fdc4e3f4c98b007f1254bd90e3900"} Nov 02 21:36:35.212618 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.212443 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6" event={"ID":"49d7528a-f6fa-49f0-97a7-461066310602","Type":"ContainerDied","Data":"1c076342f3c5e04f56f41e333639a9d71baaff67a50da68c90d2a15dd2b85d5d"} Nov 02 21:36:35.212618 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.212467 4935 scope.go:122] "RemoveContainer" containerID="94e12b06d4c1e7e110bbdaf8d16b44cc115fdc4e3f4c98b007f1254bd90e3900" Nov 02 21:36:35.214083 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.214058 4935 generic.go:356] "Generic (PLEG): container finished" podID="83fbff2a-a71b-40bd-a636-58c762e22aee" containerID="361105a3c6d5b601fdc89e15c2faee4a4a856032b88b3452ce8c32b8d0b6982f" exitCode=0 Nov 02 21:36:35.214246 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.214110 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" event={"ID":"83fbff2a-a71b-40bd-a636-58c762e22aee","Type":"ContainerDied","Data":"361105a3c6d5b601fdc89e15c2faee4a4a856032b88b3452ce8c32b8d0b6982f"} Nov 02 21:36:35.214246 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.214194 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" event={"ID":"83fbff2a-a71b-40bd-a636-58c762e22aee","Type":"ContainerDied","Data":"f38bb92b6e9b71c309d4530dd17275711ff41f4d3529d5979bd8d71974af843d"} Nov 02 21:36:35.214382 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.214264 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn" Nov 02 21:36:35.223794 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.223772 4935 scope.go:122] "RemoveContainer" containerID="94e12b06d4c1e7e110bbdaf8d16b44cc115fdc4e3f4c98b007f1254bd90e3900" Nov 02 21:36:35.224118 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:36:35.224093 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94e12b06d4c1e7e110bbdaf8d16b44cc115fdc4e3f4c98b007f1254bd90e3900\": container with ID starting with 94e12b06d4c1e7e110bbdaf8d16b44cc115fdc4e3f4c98b007f1254bd90e3900 not found: ID does not exist" containerID="94e12b06d4c1e7e110bbdaf8d16b44cc115fdc4e3f4c98b007f1254bd90e3900" Nov 02 21:36:35.224232 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.224142 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94e12b06d4c1e7e110bbdaf8d16b44cc115fdc4e3f4c98b007f1254bd90e3900"} err="failed to get container status \"94e12b06d4c1e7e110bbdaf8d16b44cc115fdc4e3f4c98b007f1254bd90e3900\": rpc error: code = NotFound desc = could not find container \"94e12b06d4c1e7e110bbdaf8d16b44cc115fdc4e3f4c98b007f1254bd90e3900\": container with ID starting with 94e12b06d4c1e7e110bbdaf8d16b44cc115fdc4e3f4c98b007f1254bd90e3900 not found: ID does not exist" Nov 02 21:36:35.224232 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.224164 4935 scope.go:122] "RemoveContainer" containerID="361105a3c6d5b601fdc89e15c2faee4a4a856032b88b3452ce8c32b8d0b6982f" Nov 02 21:36:35.236301 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.236274 4935 scope.go:122] "RemoveContainer" containerID="361105a3c6d5b601fdc89e15c2faee4a4a856032b88b3452ce8c32b8d0b6982f" Nov 02 21:36:35.236596 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:36:35.236575 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"361105a3c6d5b601fdc89e15c2faee4a4a856032b88b3452ce8c32b8d0b6982f\": container with ID starting with 361105a3c6d5b601fdc89e15c2faee4a4a856032b88b3452ce8c32b8d0b6982f not found: ID does not exist" containerID="361105a3c6d5b601fdc89e15c2faee4a4a856032b88b3452ce8c32b8d0b6982f" Nov 02 21:36:35.236668 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.236603 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361105a3c6d5b601fdc89e15c2faee4a4a856032b88b3452ce8c32b8d0b6982f"} err="failed to get container status \"361105a3c6d5b601fdc89e15c2faee4a4a856032b88b3452ce8c32b8d0b6982f\": rpc error: code = NotFound desc = could not find container \"361105a3c6d5b601fdc89e15c2faee4a4a856032b88b3452ce8c32b8d0b6982f\": container with ID starting with 361105a3c6d5b601fdc89e15c2faee4a4a856032b88b3452ce8c32b8d0b6982f not found: ID does not exist" Nov 02 21:36:35.355887 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.355853 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn"] Nov 02 21:36:35.362799 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.361976 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-nbdwn"] Nov 02 21:36:35.460917 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.460840 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6"] Nov 02 21:36:35.465649 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.465620 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-4vqc6"] Nov 02 21:36:35.779150 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.779103 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49d7528a-f6fa-49f0-97a7-461066310602" path="/var/lib/kubelet/pods/49d7528a-f6fa-49f0-97a7-461066310602/volumes" Nov 02 21:36:35.779610 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:35.779594 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83fbff2a-a71b-40bd-a636-58c762e22aee" path="/var/lib/kubelet/pods/83fbff2a-a71b-40bd-a636-58c762e22aee/volumes" Nov 02 21:36:36.698492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:36.698395 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:36.913283 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:36.913229 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:38.279598 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:38.279553 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:38.796150 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:38.792152 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:39.364632 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:39.364592 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:39.691398 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:39.691311 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:40.323096 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:40.323059 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:40.446157 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:40.443405 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r"] Nov 02 21:36:40.446157 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:40.443658 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" podUID="a290387e-9211-4298-a795-ac80d10f5ae9" containerName="istio-proxy" containerID="cri-o://de7017ec34a6a711a4cc568aaacbe96b0e200582722d52e4447890d55d30d749" gracePeriod=30 Nov 02 21:36:40.782046 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:40.782008 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:44.957076 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:44.957044 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:45.175806 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.175773 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:45.382330 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.382300 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:45.719024 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.718993 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:36:45.822337 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.822301 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-workload-socket\") pod \"a290387e-9211-4298-a795-ac80d10f5ae9\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " Nov 02 21:36:45.822499 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.822354 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a290387e-9211-4298-a795-ac80d10f5ae9-istio-token\" (UniqueName: \"kubernetes.io/projected/a290387e-9211-4298-a795-ac80d10f5ae9-istio-token\") pod \"a290387e-9211-4298-a795-ac80d10f5ae9\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " Nov 02 21:36:45.822499 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.822412 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a290387e-9211-4298-a795-ac80d10f5ae9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a290387e-9211-4298-a795-ac80d10f5ae9-istio-podinfo\") pod \"a290387e-9211-4298-a795-ac80d10f5ae9\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " Nov 02 21:36:45.822499 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.822435 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a290387e-9211-4298-a795-ac80d10f5ae9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a290387e-9211-4298-a795-ac80d10f5ae9-istiod-ca-cert\") pod \"a290387e-9211-4298-a795-ac80d10f5ae9\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " Nov 02 21:36:45.822499 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.822491 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-istio-data\") pod \"a290387e-9211-4298-a795-ac80d10f5ae9\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " Nov 02 21:36:45.822721 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.822552 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a290387e-9211-4298-a795-ac80d10f5ae9-kube-api-access-mzn7x\" (UniqueName: \"kubernetes.io/projected/a290387e-9211-4298-a795-ac80d10f5ae9-kube-api-access-mzn7x\") pod \"a290387e-9211-4298-a795-ac80d10f5ae9\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " Nov 02 21:36:45.822721 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.822579 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-workload-socket" pod "a290387e-9211-4298-a795-ac80d10f5ae9" (UID: "a290387e-9211-4298-a795-ac80d10f5ae9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:36:45.822721 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.822672 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-istio-envoy\") pod \"a290387e-9211-4298-a795-ac80d10f5ae9\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " Nov 02 21:36:45.822878 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.822738 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-credential-socket\") pod \"a290387e-9211-4298-a795-ac80d10f5ae9\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " Nov 02 21:36:45.822878 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.822805 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-workload-certs\") pod \"a290387e-9211-4298-a795-ac80d10f5ae9\" (UID: \"a290387e-9211-4298-a795-ac80d10f5ae9\") " Nov 02 21:36:45.822878 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.822836 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-istio-data" pod "a290387e-9211-4298-a795-ac80d10f5ae9" (UID: "a290387e-9211-4298-a795-ac80d10f5ae9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:36:45.823021 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.822941 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a290387e-9211-4298-a795-ac80d10f5ae9-istiod-ca-cert" pod "a290387e-9211-4298-a795-ac80d10f5ae9" (UID: "a290387e-9211-4298-a795-ac80d10f5ae9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:36:45.823213 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.823143 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-credential-socket" pod "a290387e-9211-4298-a795-ac80d10f5ae9" (UID: "a290387e-9211-4298-a795-ac80d10f5ae9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:36:45.823329 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.823277 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-workload-certs" pod "a290387e-9211-4298-a795-ac80d10f5ae9" (UID: "a290387e-9211-4298-a795-ac80d10f5ae9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:36:45.823460 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.823443 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:45.823524 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.823465 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:45.823524 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.823482 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a290387e-9211-4298-a795-ac80d10f5ae9-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:45.823524 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.823496 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:45.823524 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.823510 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:45.824788 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.824769 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a290387e-9211-4298-a795-ac80d10f5ae9-istio-token" pod "a290387e-9211-4298-a795-ac80d10f5ae9" (UID: "a290387e-9211-4298-a795-ac80d10f5ae9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:36:45.824879 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.824787 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-istio-envoy" pod "a290387e-9211-4298-a795-ac80d10f5ae9" (UID: "a290387e-9211-4298-a795-ac80d10f5ae9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:36:45.825007 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.824990 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a290387e-9211-4298-a795-ac80d10f5ae9-istio-podinfo" pod "a290387e-9211-4298-a795-ac80d10f5ae9" (UID: "a290387e-9211-4298-a795-ac80d10f5ae9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:36:45.825213 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.825192 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a290387e-9211-4298-a795-ac80d10f5ae9-kube-api-access-mzn7x" pod "a290387e-9211-4298-a795-ac80d10f5ae9" (UID: "a290387e-9211-4298-a795-ac80d10f5ae9"). InnerVolumeSpecName "kube-api-access-mzn7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:36:45.926174 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.925340 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a290387e-9211-4298-a795-ac80d10f5ae9-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:45.926174 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.925379 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a290387e-9211-4298-a795-ac80d10f5ae9-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:45.926174 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.925395 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a290387e-9211-4298-a795-ac80d10f5ae9-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:45.926174 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:45.925411 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzn7x\" (UniqueName: \"kubernetes.io/projected/a290387e-9211-4298-a795-ac80d10f5ae9-kube-api-access-mzn7x\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:36:46.366407 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:46.366366 4935 generic.go:356] "Generic (PLEG): container finished" podID="a290387e-9211-4298-a795-ac80d10f5ae9" containerID="de7017ec34a6a711a4cc568aaacbe96b0e200582722d52e4447890d55d30d749" exitCode=0 Nov 02 21:36:46.366944 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:46.366430 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" Nov 02 21:36:46.366944 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:46.366451 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" event={"ID":"a290387e-9211-4298-a795-ac80d10f5ae9","Type":"ContainerDied","Data":"de7017ec34a6a711a4cc568aaacbe96b0e200582722d52e4447890d55d30d749"} Nov 02 21:36:46.366944 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:46.366532 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r" event={"ID":"a290387e-9211-4298-a795-ac80d10f5ae9","Type":"ContainerDied","Data":"a45533ac57c91fa4996e1d335bcd0599b3c8ba8cd4cbc43496dc0483daa1a6c8"} Nov 02 21:36:46.366944 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:46.366563 4935 scope.go:122] "RemoveContainer" containerID="de7017ec34a6a711a4cc568aaacbe96b0e200582722d52e4447890d55d30d749" Nov 02 21:36:46.377679 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:46.377664 4935 scope.go:122] "RemoveContainer" containerID="de7017ec34a6a711a4cc568aaacbe96b0e200582722d52e4447890d55d30d749" Nov 02 21:36:46.377965 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:36:46.377947 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de7017ec34a6a711a4cc568aaacbe96b0e200582722d52e4447890d55d30d749\": container with ID starting with de7017ec34a6a711a4cc568aaacbe96b0e200582722d52e4447890d55d30d749 not found: ID does not exist" containerID="de7017ec34a6a711a4cc568aaacbe96b0e200582722d52e4447890d55d30d749" Nov 02 21:36:46.378023 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:46.377974 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de7017ec34a6a711a4cc568aaacbe96b0e200582722d52e4447890d55d30d749"} err="failed to get container status \"de7017ec34a6a711a4cc568aaacbe96b0e200582722d52e4447890d55d30d749\": rpc error: code = NotFound desc = could not find container \"de7017ec34a6a711a4cc568aaacbe96b0e200582722d52e4447890d55d30d749\": container with ID starting with de7017ec34a6a711a4cc568aaacbe96b0e200582722d52e4447890d55d30d749 not found: ID does not exist" Nov 02 21:36:46.520466 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:46.520426 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r"] Nov 02 21:36:46.566819 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:46.566786 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-rwh6r"] Nov 02 21:36:47.785393 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:47.785356 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a290387e-9211-4298-a795-ac80d10f5ae9" path="/var/lib/kubelet/pods/a290387e-9211-4298-a795-ac80d10f5ae9/volumes" Nov 02 21:36:48.253630 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:48.253504 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:48.467218 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:48.467179 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:50.518492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:50.518461 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:55.950732 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:36:55.950692 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:36:58.475644 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:36:58.475599 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:37:00.902720 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:00.902680 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:01.159044 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:01.158953 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:06.161193 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:06.161126 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:10.787019 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.786974 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc"] Nov 02 21:37:10.787514 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787499 4935 cpu_manager.go:414] "containerMap: removing container" podUID="49d7528a-f6fa-49f0-97a7-461066310602" containerName="istio-proxy" Nov 02 21:37:10.787558 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787514 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49d7528a-f6fa-49f0-97a7-461066310602" containerName="istio-proxy" Nov 02 21:37:10.787558 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787526 4935 cpu_manager.go:414] "containerMap: removing container" podUID="9f07e6bf-04ee-4b56-b10b-ee78424e956c" containerName="istio-proxy" Nov 02 21:37:10.787558 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787532 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f07e6bf-04ee-4b56-b10b-ee78424e956c" containerName="istio-proxy" Nov 02 21:37:10.787558 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787539 4935 cpu_manager.go:414] "containerMap: removing container" podUID="50e8e193-d60a-4ce2-add8-4e788605bd23" containerName="payload-pre-processing" Nov 02 21:37:10.787558 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787546 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50e8e193-d60a-4ce2-add8-4e788605bd23" containerName="payload-pre-processing" Nov 02 21:37:10.787709 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787560 4935 cpu_manager.go:414] "containerMap: removing container" podUID="5ba481e7-6f26-4ae3-91cf-0f2acea6fc33" containerName="istio-proxy" Nov 02 21:37:10.787709 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787566 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba481e7-6f26-4ae3-91cf-0f2acea6fc33" containerName="istio-proxy" Nov 02 21:37:10.787709 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787580 4935 cpu_manager.go:414] "containerMap: removing container" podUID="a290387e-9211-4298-a795-ac80d10f5ae9" containerName="istio-proxy" Nov 02 21:37:10.787709 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787585 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a290387e-9211-4298-a795-ac80d10f5ae9" containerName="istio-proxy" Nov 02 21:37:10.787709 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787600 4935 cpu_manager.go:414] "containerMap: removing container" podUID="b66e9095-274e-449b-b895-8c7b973b2f47" containerName="maas-api" Nov 02 21:37:10.787709 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787605 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b66e9095-274e-449b-b895-8c7b973b2f47" containerName="maas-api" Nov 02 21:37:10.787709 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787612 4935 cpu_manager.go:414] "containerMap: removing container" podUID="83fbff2a-a71b-40bd-a636-58c762e22aee" containerName="istio-proxy" Nov 02 21:37:10.787709 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787618 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83fbff2a-a71b-40bd-a636-58c762e22aee" containerName="istio-proxy" Nov 02 21:37:10.787709 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787624 4935 cpu_manager.go:414] "containerMap: removing container" podUID="d34e34f3-77b3-433c-8e39-07d64e93fb32" containerName="maas-api" Nov 02 21:37:10.787709 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787628 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d34e34f3-77b3-433c-8e39-07d64e93fb32" containerName="maas-api" Nov 02 21:37:10.787709 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787635 4935 cpu_manager.go:414] "containerMap: removing container" podUID="5df21a5e-97fc-4564-8118-884d34b0d3d1" containerName="istio-proxy" Nov 02 21:37:10.787709 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787639 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df21a5e-97fc-4564-8118-884d34b0d3d1" containerName="istio-proxy" Nov 02 21:37:10.787709 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787646 4935 cpu_manager.go:414] "containerMap: removing container" podUID="7f86cdaf-d4dd-4281-ad28-f2750dab88a9" containerName="istio-proxy" Nov 02 21:37:10.787709 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787650 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f86cdaf-d4dd-4281-ad28-f2750dab88a9" containerName="istio-proxy" Nov 02 21:37:10.787709 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787657 4935 cpu_manager.go:414] "containerMap: removing container" podUID="6a33cd12-7bf8-4b6a-af93-201a73a69240" containerName="maas-api" Nov 02 21:37:10.787709 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787661 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a33cd12-7bf8-4b6a-af93-201a73a69240" containerName="maas-api" Nov 02 21:37:10.787709 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787680 4935 cpu_manager.go:414] "containerMap: removing container" podUID="f896b6d0-edd2-4201-a17f-3671ef1517b7" containerName="istio-proxy" Nov 02 21:37:10.787709 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787685 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f896b6d0-edd2-4201-a17f-3671ef1517b7" containerName="istio-proxy" Nov 02 21:37:10.788263 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787761 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="b66e9095-274e-449b-b895-8c7b973b2f47" containerName="maas-api" Nov 02 21:37:10.788263 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787767 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="83fbff2a-a71b-40bd-a636-58c762e22aee" containerName="istio-proxy" Nov 02 21:37:10.788263 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787772 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="d34e34f3-77b3-433c-8e39-07d64e93fb32" containerName="maas-api" Nov 02 21:37:10.788263 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787778 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df21a5e-97fc-4564-8118-884d34b0d3d1" containerName="istio-proxy" Nov 02 21:37:10.788263 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787782 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f86cdaf-d4dd-4281-ad28-f2750dab88a9" containerName="istio-proxy" Nov 02 21:37:10.788263 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787787 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a33cd12-7bf8-4b6a-af93-201a73a69240" containerName="maas-api" Nov 02 21:37:10.788263 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787796 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="f896b6d0-edd2-4201-a17f-3671ef1517b7" containerName="istio-proxy" Nov 02 21:37:10.788263 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787801 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="49d7528a-f6fa-49f0-97a7-461066310602" containerName="istio-proxy" Nov 02 21:37:10.788263 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787807 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f07e6bf-04ee-4b56-b10b-ee78424e956c" containerName="istio-proxy" Nov 02 21:37:10.788263 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787812 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="50e8e193-d60a-4ce2-add8-4e788605bd23" containerName="payload-pre-processing" Nov 02 21:37:10.788263 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787818 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba481e7-6f26-4ae3-91cf-0f2acea6fc33" containerName="istio-proxy" Nov 02 21:37:10.788263 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.787825 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="a290387e-9211-4298-a795-ac80d10f5ae9" containerName="istio-proxy" Nov 02 21:37:10.791256 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.791234 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc" Nov 02 21:37:10.839835 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.839805 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-54e844-serving-cert\"" Nov 02 21:37:10.889882 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.889845 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc"] Nov 02 21:37:10.940628 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.940592 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45871915-74c7-4bf7-a338-31f16c746f76-metrics-certs\") pod \"maas-api-e2e-ipp-a-54e844-886898c54-vb7gc\" (UID: \"45871915-74c7-4bf7-a338-31f16c746f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc" Nov 02 21:37:10.940819 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.940716 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45871915-74c7-4bf7-a338-31f16c746f76-maas-api-tls\") pod \"maas-api-e2e-ipp-a-54e844-886898c54-vb7gc\" (UID: \"45871915-74c7-4bf7-a338-31f16c746f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc" Nov 02 21:37:10.940936 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.940908 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45871915-74c7-4bf7-a338-31f16c746f76-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-54e844-886898c54-vb7gc\" (UID: \"45871915-74c7-4bf7-a338-31f16c746f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc" Nov 02 21:37:10.940997 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:10.940969 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swpt7\" (UniqueName: \"kubernetes.io/projected/45871915-74c7-4bf7-a338-31f16c746f76-kube-api-access-swpt7\") pod \"maas-api-e2e-ipp-a-54e844-886898c54-vb7gc\" (UID: \"45871915-74c7-4bf7-a338-31f16c746f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc" Nov 02 21:37:11.042900 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:11.042778 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45871915-74c7-4bf7-a338-31f16c746f76-maas-api-tls\") pod \"maas-api-e2e-ipp-a-54e844-886898c54-vb7gc\" (UID: \"45871915-74c7-4bf7-a338-31f16c746f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc" Nov 02 21:37:11.043319 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:37:11.042928 4935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-serving-cert: secret "maas-api-e2e-ipp-a-54e844-serving-cert" not found Nov 02 21:37:11.043549 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:37:11.043533 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45871915-74c7-4bf7-a338-31f16c746f76-maas-api-tls podName:45871915-74c7-4bf7-a338-31f16c746f76 nodeName:}" failed. No retries permitted until 2026-11-02 21:37:11.543507393 +0000 UTC m=+1723.983082103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/45871915-74c7-4bf7-a338-31f16c746f76-maas-api-tls") pod "maas-api-e2e-ipp-a-54e844-886898c54-vb7gc" (UID: "45871915-74c7-4bf7-a338-31f16c746f76") : secret "maas-api-e2e-ipp-a-54e844-serving-cert" not found Nov 02 21:37:11.043707 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:11.043688 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45871915-74c7-4bf7-a338-31f16c746f76-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-54e844-886898c54-vb7gc\" (UID: \"45871915-74c7-4bf7-a338-31f16c746f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc" Nov 02 21:37:11.045037 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:11.045010 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swpt7\" (UniqueName: \"kubernetes.io/projected/45871915-74c7-4bf7-a338-31f16c746f76-kube-api-access-swpt7\") pod \"maas-api-e2e-ipp-a-54e844-886898c54-vb7gc\" (UID: \"45871915-74c7-4bf7-a338-31f16c746f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc" Nov 02 21:37:11.045173 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:11.044787 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45871915-74c7-4bf7-a338-31f16c746f76-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-54e844-886898c54-vb7gc\" (UID: \"45871915-74c7-4bf7-a338-31f16c746f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc" Nov 02 21:37:11.045281 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:11.045259 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45871915-74c7-4bf7-a338-31f16c746f76-metrics-certs\") pod \"maas-api-e2e-ipp-a-54e844-886898c54-vb7gc\" (UID: \"45871915-74c7-4bf7-a338-31f16c746f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc" Nov 02 21:37:11.048508 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:11.048480 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45871915-74c7-4bf7-a338-31f16c746f76-metrics-certs\") pod \"maas-api-e2e-ipp-a-54e844-886898c54-vb7gc\" (UID: \"45871915-74c7-4bf7-a338-31f16c746f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc" Nov 02 21:37:11.064990 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:11.064949 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:11.101473 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:11.101436 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swpt7\" (UniqueName: \"kubernetes.io/projected/45871915-74c7-4bf7-a338-31f16c746f76-kube-api-access-swpt7\") pod \"maas-api-e2e-ipp-a-54e844-886898c54-vb7gc\" (UID: \"45871915-74c7-4bf7-a338-31f16c746f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc" Nov 02 21:37:11.301849 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:11.301754 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:11.529593 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:11.529552 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:11.551994 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:11.551899 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45871915-74c7-4bf7-a338-31f16c746f76-maas-api-tls\") pod \"maas-api-e2e-ipp-a-54e844-886898c54-vb7gc\" (UID: \"45871915-74c7-4bf7-a338-31f16c746f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc" Nov 02 21:37:11.554484 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:11.554456 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45871915-74c7-4bf7-a338-31f16c746f76-maas-api-tls\") pod \"maas-api-e2e-ipp-a-54e844-886898c54-vb7gc\" (UID: \"45871915-74c7-4bf7-a338-31f16c746f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc" Nov 02 21:37:11.704669 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:11.704636 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc" Nov 02 21:37:12.077736 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:12.077703 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc"] Nov 02 21:37:12.078702 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:37:12.078673 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45871915_74c7_4bf7_a338_31f16c746f76.slice/crio-7cebeff73723d82ea1f45dfa70af91ae0c8157aaa67ccc3a21799669f6b3191f WatchSource:0}: Error finding container 7cebeff73723d82ea1f45dfa70af91ae0c8157aaa67ccc3a21799669f6b3191f: Status 404 returned error can't find the container with id 7cebeff73723d82ea1f45dfa70af91ae0c8157aaa67ccc3a21799669f6b3191f Nov 02 21:37:12.080223 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:12.080205 4935 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:37:12.560498 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:12.560465 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc" event={"ID":"45871915-74c7-4bf7-a338-31f16c746f76","Type":"ContainerStarted","Data":"7cebeff73723d82ea1f45dfa70af91ae0c8157aaa67ccc3a21799669f6b3191f"} Nov 02 21:37:13.568244 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:13.568211 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc" event={"ID":"45871915-74c7-4bf7-a338-31f16c746f76","Type":"ContainerStarted","Data":"3adc7d6437961cdac22534e95601415d4f220bcf0172754a3084e9f7f155d82b"} Nov 02 21:37:13.568651 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:13.568407 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc" Nov 02 21:37:13.675513 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:13.675451 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc" podStartSLOduration=2.765903024 podStartE2EDuration="3.67543647s" totalImagesPullingTime="909.533446ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:37:13.674417922 +0000 UTC m=+1726.113992644" watchObservedRunningTime="2026-11-02 21:37:13.67543647 +0000 UTC m=+1726.115011192" Nov 02 21:37:13.819205 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:13.819089 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:14.500885 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:14.500818 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:14.911119 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:14.911085 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:16.076738 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.076701 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46"] Nov 02 21:37:16.081037 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.081007 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.129784 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.129748 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-70efbbab-openshift-default-dockercfg-m2wbl\"" Nov 02 21:37:16.180515 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.180476 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46"] Nov 02 21:37:16.217554 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.217513 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-credential-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.217721 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.217692 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hkpn\" (UniqueName: \"kubernetes.io/projected/8e2fddf5-ee5d-4872-91c2-b073e5261728-kube-api-access-9hkpn\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.217810 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.217782 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-istio-data\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.217941 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.217834 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-workload-certs\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.217941 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.217870 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-istio-envoy\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.218055 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.217989 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2fddf5-ee5d-4872-91c2-b073e5261728-istiod-ca-cert\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.218055 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.218038 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-workload-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.218195 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.218175 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2fddf5-ee5d-4872-91c2-b073e5261728-istio-podinfo\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.218260 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.218205 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2fddf5-ee5d-4872-91c2-b073e5261728-istio-token\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.319756 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.319718 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-istio-data\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.319756 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.319767 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-workload-certs\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.320019 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.319799 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-istio-envoy\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.320019 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.319845 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2fddf5-ee5d-4872-91c2-b073e5261728-istiod-ca-cert\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.320019 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.319879 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-workload-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.320019 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.319941 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2fddf5-ee5d-4872-91c2-b073e5261728-istio-podinfo\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.320019 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.319968 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2fddf5-ee5d-4872-91c2-b073e5261728-istio-token\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.320289 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.320125 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-credential-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.320289 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.320199 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hkpn\" (UniqueName: \"kubernetes.io/projected/8e2fddf5-ee5d-4872-91c2-b073e5261728-kube-api-access-9hkpn\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.320819 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.320792 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2fddf5-ee5d-4872-91c2-b073e5261728-istiod-ca-cert\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.320861 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.320818 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-workload-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.321320 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.321296 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-istio-data\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.321631 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.321602 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-credential-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.321751 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.321704 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-workload-certs\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.323502 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.323481 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-istio-envoy\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.323593 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.323519 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2fddf5-ee5d-4872-91c2-b073e5261728-istio-podinfo\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.378220 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.377367 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2fddf5-ee5d-4872-91c2-b073e5261728-istio-token\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.378220 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.377420 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hkpn\" (UniqueName: \"kubernetes.io/projected/8e2fddf5-ee5d-4872-91c2-b073e5261728-kube-api-access-9hkpn\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.396659 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.396628 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:16.397707 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.397678 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:16.816152 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.816109 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46"] Nov 02 21:37:16.817311 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:37:16.817280 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e2fddf5_ee5d_4872_91c2_b073e5261728.slice/crio-3b925e1f9f30d0d54621cb580c9991362e3d4455dc82959118f7f297df486079 WatchSource:0}: Error finding container 3b925e1f9f30d0d54621cb580c9991362e3d4455dc82959118f7f297df486079: Status 404 returned error can't find the container with id 3b925e1f9f30d0d54621cb580c9991362e3d4455dc82959118f7f297df486079 Nov 02 21:37:16.819919 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.819885 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:37:16.820008 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.819953 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:37:16.820008 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:16.819984 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:37:17.609799 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:17.609763 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" event={"ID":"8e2fddf5-ee5d-4872-91c2-b073e5261728","Type":"ContainerStarted","Data":"057ae25402254f08f401e9385b4930c01105fd64974dca249305f299afb43967"} Nov 02 21:37:17.610315 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:17.609819 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" event={"ID":"8e2fddf5-ee5d-4872-91c2-b073e5261728","Type":"ContainerStarted","Data":"3b925e1f9f30d0d54621cb580c9991362e3d4455dc82959118f7f297df486079"} Nov 02 21:37:17.812585 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:17.812529 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" podStartSLOduration=1.812513696 podStartE2EDuration="1.812513696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:37:17.766072223 +0000 UTC m=+1730.205646940" watchObservedRunningTime="2026-11-02 21:37:17.812513696 +0000 UTC m=+1730.252088473" Nov 02 21:37:18.253833 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:18.253795 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:18.397394 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:18.397358 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:18.399163 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:18.399109 4935 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 21:37:18.399281 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:18.399185 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" podUID="8e2fddf5-ee5d-4872-91c2-b073e5261728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 21:37:18.470469 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:18.470428 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:19.397307 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:19.397273 4935 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 21:37:19.397797 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:19.397336 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" podUID="8e2fddf5-ee5d-4872-91c2-b073e5261728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 21:37:19.582051 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:19.582019 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc" Nov 02 21:37:20.397760 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:20.397724 4935 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 21:37:20.398200 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:20.397782 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" podUID="8e2fddf5-ee5d-4872-91c2-b073e5261728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 21:37:21.397074 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:21.397033 4935 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 21:37:21.397281 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:21.397089 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" podUID="8e2fddf5-ee5d-4872-91c2-b073e5261728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 21:37:21.457153 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:21.456146 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:21.667035 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:21.666883 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:22.013148 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:22.013069 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:22.397594 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:22.397510 4935 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 21:37:22.397594 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:22.397565 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" podUID="8e2fddf5-ee5d-4872-91c2-b073e5261728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 21:37:23.767016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:23.766984 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:23.768990 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:23.768958 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:23.784478 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:23.784443 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:37:27.207609 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:27.207574 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:27.757933 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:27.757896 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:28.470936 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:37:28.470894 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:37:31.070924 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.070885 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb"] Nov 02 21:37:31.076650 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.076628 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.171093 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.171042 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-workload-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.171270 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.171208 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-credential-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.171332 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.171264 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92x2k\" (UniqueName: \"kubernetes.io/projected/3ea341f9-33df-461e-9f33-69be35dea3ba-kube-api-access-92x2k\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.171418 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.171381 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-workload-certs\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.171558 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.171477 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ea341f9-33df-461e-9f33-69be35dea3ba-istiod-ca-cert\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.171558 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.171508 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ea341f9-33df-461e-9f33-69be35dea3ba-istio-podinfo\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.171681 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.171604 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ea341f9-33df-461e-9f33-69be35dea3ba-istio-token\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.171681 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.171645 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-istio-data\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.171785 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.171701 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-istio-envoy\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.176853 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.176818 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb"] Nov 02 21:37:31.273308 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.273271 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-workload-certs\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.273517 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.273352 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ea341f9-33df-461e-9f33-69be35dea3ba-istiod-ca-cert\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.273517 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.273377 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ea341f9-33df-461e-9f33-69be35dea3ba-istio-podinfo\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.273517 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.273454 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ea341f9-33df-461e-9f33-69be35dea3ba-istio-token\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.273517 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.273483 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-istio-data\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.273742 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.273538 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-istio-envoy\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.273742 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.273575 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-workload-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.273742 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.273647 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-credential-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.273742 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.273695 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92x2k\" (UniqueName: \"kubernetes.io/projected/3ea341f9-33df-461e-9f33-69be35dea3ba-kube-api-access-92x2k\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.273971 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.273734 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-workload-certs\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.273971 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.273885 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-istio-data\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.274109 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.274081 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-credential-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.274109 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.274103 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-workload-socket\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.274253 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.274204 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ea341f9-33df-461e-9f33-69be35dea3ba-istiod-ca-cert\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.276193 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.276165 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-istio-envoy\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.276760 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.276740 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ea341f9-33df-461e-9f33-69be35dea3ba-istio-podinfo\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.293167 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.292918 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:31.328225 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.328084 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92x2k\" (UniqueName: \"kubernetes.io/projected/3ea341f9-33df-461e-9f33-69be35dea3ba-kube-api-access-92x2k\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.329905 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.329855 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ea341f9-33df-461e-9f33-69be35dea3ba-istio-token\") pod \"e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.390482 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.390450 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:31.828540 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.828500 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb"] Nov 02 21:37:31.866984 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:37:31.866952 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea341f9_33df_461e_9f33_69be35dea3ba.slice/crio-d2415e5d5e3b190fd2854ae22f823e6a133002fee180e7a3f10fd1bdc70b48f8 WatchSource:0}: Error finding container d2415e5d5e3b190fd2854ae22f823e6a133002fee180e7a3f10fd1bdc70b48f8: Status 404 returned error can't find the container with id d2415e5d5e3b190fd2854ae22f823e6a133002fee180e7a3f10fd1bdc70b48f8 Nov 02 21:37:31.869843 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.869808 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:37:31.869983 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.869926 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:37:31.869983 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:31.869974 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:37:32.418367 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:32.418327 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:32.781075 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:32.781035 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" event={"ID":"3ea341f9-33df-461e-9f33-69be35dea3ba","Type":"ContainerStarted","Data":"5ee6104d02815b02ef1af01db00f3317d180091cddd1a27e44044f06f1eef0ae"} Nov 02 21:37:32.781255 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:32.781099 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" event={"ID":"3ea341f9-33df-461e-9f33-69be35dea3ba","Type":"ContainerStarted","Data":"d2415e5d5e3b190fd2854ae22f823e6a133002fee180e7a3f10fd1bdc70b48f8"} Nov 02 21:37:32.900465 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:32.900406 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" podStartSLOduration=1.9003928289999998 podStartE2EDuration="1.900392829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:37:32.897278097 +0000 UTC m=+1745.336852820" watchObservedRunningTime="2026-11-02 21:37:32.900392829 +0000 UTC m=+1745.339967551" Nov 02 21:37:33.209790 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:33.209688 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:33.391325 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:33.391290 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:33.393119 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:33.393089 4935 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 21:37:33.393251 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:33.393167 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" podUID="3ea341f9-33df-461e-9f33-69be35dea3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 21:37:34.391255 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:34.391215 4935 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 21:37:34.391779 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:34.391274 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" podUID="3ea341f9-33df-461e-9f33-69be35dea3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 21:37:35.391215 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:35.390943 4935 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 21:37:35.391215 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:35.391006 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" podUID="3ea341f9-33df-461e-9f33-69be35dea3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 21:37:35.679580 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:35.679485 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:36.391211 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:36.391171 4935 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 21:37:36.391392 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:36.391235 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" podUID="3ea341f9-33df-461e-9f33-69be35dea3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 21:37:37.391909 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:37.391867 4935 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 21:37:37.392358 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:37.391924 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" podUID="3ea341f9-33df-461e-9f33-69be35dea3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 21:37:38.338969 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:38.338880 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9"] Nov 02 21:37:38.343577 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:38.343544 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9" Nov 02 21:37:38.392461 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:38.392422 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-15253e-serving-cert\"" Nov 02 21:37:38.457122 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:38.457087 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9"] Nov 02 21:37:38.458933 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:38.458876 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:38.499098 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:38.499056 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xl7k\" (UniqueName: \"kubernetes.io/projected/e29452f7-d10d-4157-9789-f98c9111d8e8-kube-api-access-5xl7k\") pod \"maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9\" (UID: \"e29452f7-d10d-4157-9789-f98c9111d8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9" Nov 02 21:37:38.499361 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:38.499342 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e29452f7-d10d-4157-9789-f98c9111d8e8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9\" (UID: \"e29452f7-d10d-4157-9789-f98c9111d8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9" Nov 02 21:37:38.499479 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:38.499464 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e29452f7-d10d-4157-9789-f98c9111d8e8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9\" (UID: \"e29452f7-d10d-4157-9789-f98c9111d8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9" Nov 02 21:37:38.499663 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:38.499649 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e29452f7-d10d-4157-9789-f98c9111d8e8-metrics-certs\") pod \"maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9\" (UID: \"e29452f7-d10d-4157-9789-f98c9111d8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9" Nov 02 21:37:38.577787 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:38.577755 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:38.579707 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:38.579683 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:38.583738 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:38.583713 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:37:38.601370 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:38.601224 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xl7k\" (UniqueName: \"kubernetes.io/projected/e29452f7-d10d-4157-9789-f98c9111d8e8-kube-api-access-5xl7k\") pod \"maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9\" (UID: \"e29452f7-d10d-4157-9789-f98c9111d8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9" Nov 02 21:37:38.602065 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:38.602039 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e29452f7-d10d-4157-9789-f98c9111d8e8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9\" (UID: \"e29452f7-d10d-4157-9789-f98c9111d8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9" Nov 02 21:37:38.602332 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:38.602311 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e29452f7-d10d-4157-9789-f98c9111d8e8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9\" (UID: \"e29452f7-d10d-4157-9789-f98c9111d8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9" Nov 02 21:37:38.609780 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:38.609754 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e29452f7-d10d-4157-9789-f98c9111d8e8-metrics-certs\") pod \"maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9\" (UID: \"e29452f7-d10d-4157-9789-f98c9111d8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9" Nov 02 21:37:38.610095 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:38.603468 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e29452f7-d10d-4157-9789-f98c9111d8e8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9\" (UID: \"e29452f7-d10d-4157-9789-f98c9111d8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9" Nov 02 21:37:38.610233 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:37:38.602722 4935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-serving-cert: secret "maas-api-e2e-ipp-a-15253e-serving-cert" not found Nov 02 21:37:38.610372 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:37:38.610360 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e29452f7-d10d-4157-9789-f98c9111d8e8-maas-api-tls podName:e29452f7-d10d-4157-9789-f98c9111d8e8 nodeName:}" failed. No retries permitted until 2026-11-02 21:37:39.1103357 +0000 UTC m=+1751.549910415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e29452f7-d10d-4157-9789-f98c9111d8e8-maas-api-tls") pod "maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9" (UID: "e29452f7-d10d-4157-9789-f98c9111d8e8") : secret "maas-api-e2e-ipp-a-15253e-serving-cert" not found Nov 02 21:37:38.623780 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:38.623745 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e29452f7-d10d-4157-9789-f98c9111d8e8-metrics-certs\") pod \"maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9\" (UID: \"e29452f7-d10d-4157-9789-f98c9111d8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9" Nov 02 21:37:38.657338 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:38.657299 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xl7k\" (UniqueName: \"kubernetes.io/projected/e29452f7-d10d-4157-9789-f98c9111d8e8-kube-api-access-5xl7k\") pod \"maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9\" (UID: \"e29452f7-d10d-4157-9789-f98c9111d8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9" Nov 02 21:37:39.117604 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:39.117569 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e29452f7-d10d-4157-9789-f98c9111d8e8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9\" (UID: \"e29452f7-d10d-4157-9789-f98c9111d8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9" Nov 02 21:37:39.120063 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:39.120031 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e29452f7-d10d-4157-9789-f98c9111d8e8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9\" (UID: \"e29452f7-d10d-4157-9789-f98c9111d8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9" Nov 02 21:37:39.260991 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:39.260952 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9" Nov 02 21:37:39.631080 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:39.631053 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9"] Nov 02 21:37:39.634227 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:37:39.634196 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29452f7_d10d_4157_9789_f98c9111d8e8.slice/crio-529bc89288835fb8c122c633e100c2bc6fd1f2211801c01fa1480b4b7b6b24db WatchSource:0}: Error finding container 529bc89288835fb8c122c633e100c2bc6fd1f2211801c01fa1480b4b7b6b24db: Status 404 returned error can't find the container with id 529bc89288835fb8c122c633e100c2bc6fd1f2211801c01fa1480b4b7b6b24db Nov 02 21:37:39.830489 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:39.830453 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9" event={"ID":"e29452f7-d10d-4157-9789-f98c9111d8e8","Type":"ContainerStarted","Data":"529bc89288835fb8c122c633e100c2bc6fd1f2211801c01fa1480b4b7b6b24db"} Nov 02 21:37:40.559246 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:40.559218 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc"] Nov 02 21:37:40.839876 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:40.839840 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9" event={"ID":"e29452f7-d10d-4157-9789-f98c9111d8e8","Type":"ContainerStarted","Data":"7466d7cbb210860dceb1a0a567310986878792a98cb5d4dfa71f869f1df17396"} Nov 02 21:37:40.840411 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:40.839983 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9" Nov 02 21:37:40.951437 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:40.951385 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9" podStartSLOduration=2.100419711 podStartE2EDuration="2.951370262s" totalImagesPullingTime="850.950551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:37:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:37:40.947487455 +0000 UTC m=+1753.387062178" watchObservedRunningTime="2026-11-02 21:37:40.951370262 +0000 UTC m=+1753.390944989" Nov 02 21:37:40.995221 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:40.995120 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:41.646381 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:41.646346 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d"] Nov 02 21:37:41.964703 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:41.964619 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:42.185455 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:42.185425 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:44.170009 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:44.169980 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:44.467284 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:44.467180 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:46.646999 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:46.646953 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d" podUID="4efcdacc-c172-4f9a-b877-3b0a510252de" containerName="payload-pre-processing" containerID="cri-o://3724f4c45127d5a011ea69697b363b4b24a7ac03c3b08abcc68b64ce5b085b38" gracePeriod=25 Nov 02 21:37:46.855555 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:46.855520 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9" Nov 02 21:37:46.889767 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:46.889730 4935 generic.go:356] "Generic (PLEG): container finished" podID="4efcdacc-c172-4f9a-b877-3b0a510252de" containerID="3724f4c45127d5a011ea69697b363b4b24a7ac03c3b08abcc68b64ce5b085b38" exitCode=0 Nov 02 21:37:46.889893 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:46.889793 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d" event={"ID":"4efcdacc-c172-4f9a-b877-3b0a510252de","Type":"ContainerDied","Data":"3724f4c45127d5a011ea69697b363b4b24a7ac03c3b08abcc68b64ce5b085b38"} Nov 02 21:37:46.929024 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:46.928991 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d" Nov 02 21:37:47.039700 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:47.039670 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4efcdacc-c172-4f9a-b877-3b0a510252de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4efcdacc-c172-4f9a-b877-3b0a510252de-plugins-config-volume\") pod \"4efcdacc-c172-4f9a-b877-3b0a510252de\" (UID: \"4efcdacc-c172-4f9a-b877-3b0a510252de\") " Nov 02 21:37:47.039889 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:47.039811 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4efcdacc-c172-4f9a-b877-3b0a510252de-kube-api-access-6v9mx\" (UniqueName: \"kubernetes.io/projected/4efcdacc-c172-4f9a-b877-3b0a510252de-kube-api-access-6v9mx\") pod \"4efcdacc-c172-4f9a-b877-3b0a510252de\" (UID: \"4efcdacc-c172-4f9a-b877-3b0a510252de\") " Nov 02 21:37:47.040150 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:47.040110 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4efcdacc-c172-4f9a-b877-3b0a510252de-plugins-config-volume" pod "4efcdacc-c172-4f9a-b877-3b0a510252de" (UID: "4efcdacc-c172-4f9a-b877-3b0a510252de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:37:47.040288 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:47.040268 4935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4efcdacc-c172-4f9a-b877-3b0a510252de-plugins-config-volume\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:47.042060 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:47.042029 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4efcdacc-c172-4f9a-b877-3b0a510252de-kube-api-access-6v9mx" pod "4efcdacc-c172-4f9a-b877-3b0a510252de" (UID: "4efcdacc-c172-4f9a-b877-3b0a510252de"). InnerVolumeSpecName "kube-api-access-6v9mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:37:47.141305 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:47.141265 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v9mx\" (UniqueName: \"kubernetes.io/projected/4efcdacc-c172-4f9a-b877-3b0a510252de-kube-api-access-6v9mx\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:47.899091 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:47.899054 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d" event={"ID":"4efcdacc-c172-4f9a-b877-3b0a510252de","Type":"ContainerDied","Data":"cfcd5c1fc21dbfc109ffb24bf5f98a519c61524a1313779972ae659098017732"} Nov 02 21:37:47.899091 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:47.899075 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d" Nov 02 21:37:47.899611 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:47.899115 4935 scope.go:122] "RemoveContainer" containerID="3724f4c45127d5a011ea69697b363b4b24a7ac03c3b08abcc68b64ce5b085b38" Nov 02 21:37:48.054373 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:48.054295 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d"] Nov 02 21:37:48.100870 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:48.100838 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-682116-65896df67f-shl8d"] Nov 02 21:37:49.718596 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:49.718558 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:49.778590 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:49.778557 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4efcdacc-c172-4f9a-b877-3b0a510252de" path="/var/lib/kubelet/pods/4efcdacc-c172-4f9a-b877-3b0a510252de/volumes" Nov 02 21:37:49.964602 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:49.964564 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:55.227926 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:55.227886 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:55.580581 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:55.580523 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" podUID="f58a753c-53b9-4570-8875-6123f00fed56" containerName="main" containerID="cri-o://e1a1d1290e970cd9dcd8e73aab18beabb3d086f3c391f21b6553e68bc5cb33f8" gracePeriod=45 Nov 02 21:37:55.604065 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:55.604035 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" podUID="f58a753c-53b9-4570-8875-6123f00fed56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": dial tcp 10.132.0.83:8000: connect: connection refused" Nov 02 21:37:55.861917 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:55.861893 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" Nov 02 21:37:55.955221 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:55.955190 4935 generic.go:356] "Generic (PLEG): container finished" podID="f58a753c-53b9-4570-8875-6123f00fed56" containerID="e1a1d1290e970cd9dcd8e73aab18beabb3d086f3c391f21b6553e68bc5cb33f8" exitCode=0 Nov 02 21:37:55.955394 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:55.955241 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" Nov 02 21:37:55.955394 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:55.955280 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" event={"ID":"f58a753c-53b9-4570-8875-6123f00fed56","Type":"ContainerDied","Data":"e1a1d1290e970cd9dcd8e73aab18beabb3d086f3c391f21b6553e68bc5cb33f8"} Nov 02 21:37:55.955394 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:55.955331 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc" event={"ID":"f58a753c-53b9-4570-8875-6123f00fed56","Type":"ContainerDied","Data":"d550a5c9e7da9b37ac176154e91fd187a41b007fe67cfb91498b310fcabe7503"} Nov 02 21:37:55.955394 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:55.955351 4935 scope.go:122] "RemoveContainer" containerID="e1a1d1290e970cd9dcd8e73aab18beabb3d086f3c391f21b6553e68bc5cb33f8" Nov 02 21:37:55.964888 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:55.964870 4935 scope.go:122] "RemoveContainer" containerID="e1a1d1290e970cd9dcd8e73aab18beabb3d086f3c391f21b6553e68bc5cb33f8" Nov 02 21:37:55.965175 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:37:55.965155 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1a1d1290e970cd9dcd8e73aab18beabb3d086f3c391f21b6553e68bc5cb33f8\": container with ID starting with e1a1d1290e970cd9dcd8e73aab18beabb3d086f3c391f21b6553e68bc5cb33f8 not found: ID does not exist" containerID="e1a1d1290e970cd9dcd8e73aab18beabb3d086f3c391f21b6553e68bc5cb33f8" Nov 02 21:37:55.965277 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:55.965185 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1a1d1290e970cd9dcd8e73aab18beabb3d086f3c391f21b6553e68bc5cb33f8"} err="failed to get container status \"e1a1d1290e970cd9dcd8e73aab18beabb3d086f3c391f21b6553e68bc5cb33f8\": rpc error: code = NotFound desc = could not find container \"e1a1d1290e970cd9dcd8e73aab18beabb3d086f3c391f21b6553e68bc5cb33f8\": container with ID starting with e1a1d1290e970cd9dcd8e73aab18beabb3d086f3c391f21b6553e68bc5cb33f8 not found: ID does not exist" Nov 02 21:37:55.965277 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:55.965163 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58a753c-53b9-4570-8875-6123f00fed56-kube-api-access-nstfd\" (UniqueName: \"kubernetes.io/projected/f58a753c-53b9-4570-8875-6123f00fed56-kube-api-access-nstfd\") pod \"f58a753c-53b9-4570-8875-6123f00fed56\" (UID: \"f58a753c-53b9-4570-8875-6123f00fed56\") " Nov 02 21:37:55.965416 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:55.965310 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f58a753c-53b9-4570-8875-6123f00fed56-tls-certs\" (UniqueName: \"kubernetes.io/secret/f58a753c-53b9-4570-8875-6123f00fed56-tls-certs\") pod \"f58a753c-53b9-4570-8875-6123f00fed56\" (UID: \"f58a753c-53b9-4570-8875-6123f00fed56\") " Nov 02 21:37:55.965416 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:55.965406 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58a753c-53b9-4570-8875-6123f00fed56-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f58a753c-53b9-4570-8875-6123f00fed56-dshm\") pod \"f58a753c-53b9-4570-8875-6123f00fed56\" (UID: \"f58a753c-53b9-4570-8875-6123f00fed56\") " Nov 02 21:37:55.965513 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:55.965494 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58a753c-53b9-4570-8875-6123f00fed56-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f58a753c-53b9-4570-8875-6123f00fed56-tmp-dir\") pod \"f58a753c-53b9-4570-8875-6123f00fed56\" (UID: \"f58a753c-53b9-4570-8875-6123f00fed56\") " Nov 02 21:37:55.965576 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:55.965560 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58a753c-53b9-4570-8875-6123f00fed56-home\" (UniqueName: \"kubernetes.io/empty-dir/f58a753c-53b9-4570-8875-6123f00fed56-home\") pod \"f58a753c-53b9-4570-8875-6123f00fed56\" (UID: \"f58a753c-53b9-4570-8875-6123f00fed56\") " Nov 02 21:37:55.965646 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:55.965632 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58a753c-53b9-4570-8875-6123f00fed56-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f58a753c-53b9-4570-8875-6123f00fed56-model-cache\") pod \"f58a753c-53b9-4570-8875-6123f00fed56\" (UID: \"f58a753c-53b9-4570-8875-6123f00fed56\") " Nov 02 21:37:55.965845 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:55.965817 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58a753c-53b9-4570-8875-6123f00fed56-tmp-dir" pod "f58a753c-53b9-4570-8875-6123f00fed56" (UID: "f58a753c-53b9-4570-8875-6123f00fed56"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:37:55.965922 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:55.965851 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58a753c-53b9-4570-8875-6123f00fed56-home" pod "f58a753c-53b9-4570-8875-6123f00fed56" (UID: "f58a753c-53b9-4570-8875-6123f00fed56"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:37:55.966096 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:55.966066 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58a753c-53b9-4570-8875-6123f00fed56-model-cache" pod "f58a753c-53b9-4570-8875-6123f00fed56" (UID: "f58a753c-53b9-4570-8875-6123f00fed56"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:37:55.966250 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:55.966216 4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f58a753c-53b9-4570-8875-6123f00fed56-model-cache\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:55.966250 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:55.966231 4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f58a753c-53b9-4570-8875-6123f00fed56-tmp-dir\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:55.966250 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:55.966249 4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f58a753c-53b9-4570-8875-6123f00fed56-home\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:55.967455 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:55.967433 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58a753c-53b9-4570-8875-6123f00fed56-kube-api-access-nstfd" pod "f58a753c-53b9-4570-8875-6123f00fed56" (UID: "f58a753c-53b9-4570-8875-6123f00fed56"). InnerVolumeSpecName "kube-api-access-nstfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:37:55.967455 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:55.967446 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58a753c-53b9-4570-8875-6123f00fed56-dshm" pod "f58a753c-53b9-4570-8875-6123f00fed56" (UID: "f58a753c-53b9-4570-8875-6123f00fed56"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:37:55.967588 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:55.967573 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f58a753c-53b9-4570-8875-6123f00fed56-tls-certs" pod "f58a753c-53b9-4570-8875-6123f00fed56" (UID: "f58a753c-53b9-4570-8875-6123f00fed56"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:37:56.067550 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:56.067511 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nstfd\" (UniqueName: \"kubernetes.io/projected/f58a753c-53b9-4570-8875-6123f00fed56-kube-api-access-nstfd\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:56.067718 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:56.067557 4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f58a753c-53b9-4570-8875-6123f00fed56-tls-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:56.067718 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:56.067574 4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f58a753c-53b9-4570-8875-6123f00fed56-dshm\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:56.370037 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:56.370002 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc"] Nov 02 21:37:56.416718 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:56.416688 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-70efbbab/e2e-lifecycle-model-70efbbab-kserve-54f7b9c8c7-lg7vc"] Nov 02 21:37:57.779894 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:57.779852 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58a753c-53b9-4570-8875-6123f00fed56" path="/var/lib/kubelet/pods/f58a753c-53b9-4570-8875-6123f00fed56/volumes" Nov 02 21:37:58.470502 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:37:58.470466 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:37:59.462257 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:59.462216 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq"] Nov 02 21:37:59.462749 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:59.462541 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq" podUID="aa406b90-6571-4f44-a9ba-b3299863e736" containerName="maas-api" containerID="cri-o://a8e3baecc27c1cd088d455a47b3231c90ef456655e1057400cd1d77c7bf9f02c" gracePeriod=30 Nov 02 21:37:59.642153 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:59.638643 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:37:59.813937 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:59.813909 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq" Nov 02 21:37:59.922920 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:59.922881 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa406b90-6571-4f44-a9ba-b3299863e736-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa406b90-6571-4f44-a9ba-b3299863e736-maas-api-tls\") pod \"aa406b90-6571-4f44-a9ba-b3299863e736\" (UID: \"aa406b90-6571-4f44-a9ba-b3299863e736\") " Nov 02 21:37:59.923114 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:59.922950 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa406b90-6571-4f44-a9ba-b3299863e736-kube-api-access-t7vnl\" (UniqueName: \"kubernetes.io/projected/aa406b90-6571-4f44-a9ba-b3299863e736-kube-api-access-t7vnl\") pod \"aa406b90-6571-4f44-a9ba-b3299863e736\" (UID: \"aa406b90-6571-4f44-a9ba-b3299863e736\") " Nov 02 21:37:59.923114 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:59.923012 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa406b90-6571-4f44-a9ba-b3299863e736-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa406b90-6571-4f44-a9ba-b3299863e736-metrics-certs\") pod \"aa406b90-6571-4f44-a9ba-b3299863e736\" (UID: \"aa406b90-6571-4f44-a9ba-b3299863e736\") " Nov 02 21:37:59.923114 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:59.923092 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa406b90-6571-4f44-a9ba-b3299863e736-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa406b90-6571-4f44-a9ba-b3299863e736-openshift-service-ca\") pod \"aa406b90-6571-4f44-a9ba-b3299863e736\" (UID: \"aa406b90-6571-4f44-a9ba-b3299863e736\") " Nov 02 21:37:59.925317 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:59.925284 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa406b90-6571-4f44-a9ba-b3299863e736-openshift-service-ca" pod "aa406b90-6571-4f44-a9ba-b3299863e736" (UID: "aa406b90-6571-4f44-a9ba-b3299863e736"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:37:59.943291 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:59.943259 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa406b90-6571-4f44-a9ba-b3299863e736-metrics-certs" pod "aa406b90-6571-4f44-a9ba-b3299863e736" (UID: "aa406b90-6571-4f44-a9ba-b3299863e736"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:37:59.951458 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:59.951426 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa406b90-6571-4f44-a9ba-b3299863e736-kube-api-access-t7vnl" pod "aa406b90-6571-4f44-a9ba-b3299863e736" (UID: "aa406b90-6571-4f44-a9ba-b3299863e736"). InnerVolumeSpecName "kube-api-access-t7vnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:37:59.957738 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:59.957709 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa406b90-6571-4f44-a9ba-b3299863e736-maas-api-tls" pod "aa406b90-6571-4f44-a9ba-b3299863e736" (UID: "aa406b90-6571-4f44-a9ba-b3299863e736"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:37:59.985688 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:59.985657 4935 generic.go:356] "Generic (PLEG): container finished" podID="aa406b90-6571-4f44-a9ba-b3299863e736" containerID="a8e3baecc27c1cd088d455a47b3231c90ef456655e1057400cd1d77c7bf9f02c" exitCode=0 Nov 02 21:37:59.985876 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:59.985703 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq" Nov 02 21:37:59.985876 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:59.985743 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq" event={"ID":"aa406b90-6571-4f44-a9ba-b3299863e736","Type":"ContainerDied","Data":"a8e3baecc27c1cd088d455a47b3231c90ef456655e1057400cd1d77c7bf9f02c"} Nov 02 21:37:59.985876 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:59.985806 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq" event={"ID":"aa406b90-6571-4f44-a9ba-b3299863e736","Type":"ContainerDied","Data":"88456a06d6a571e3585b72fd79fd2750261dcf376417ff7dda40e5f9c653e4bd"} Nov 02 21:37:59.985876 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:37:59.985838 4935 scope.go:122] "RemoveContainer" containerID="a8e3baecc27c1cd088d455a47b3231c90ef456655e1057400cd1d77c7bf9f02c" Nov 02 21:38:00.002688 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:00.002668 4935 scope.go:122] "RemoveContainer" containerID="a8e3baecc27c1cd088d455a47b3231c90ef456655e1057400cd1d77c7bf9f02c" Nov 02 21:38:00.002999 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:38:00.002977 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8e3baecc27c1cd088d455a47b3231c90ef456655e1057400cd1d77c7bf9f02c\": container with ID starting with a8e3baecc27c1cd088d455a47b3231c90ef456655e1057400cd1d77c7bf9f02c not found: ID does not exist" containerID="a8e3baecc27c1cd088d455a47b3231c90ef456655e1057400cd1d77c7bf9f02c" Nov 02 21:38:00.003073 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:00.003008 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8e3baecc27c1cd088d455a47b3231c90ef456655e1057400cd1d77c7bf9f02c"} err="failed to get container status \"a8e3baecc27c1cd088d455a47b3231c90ef456655e1057400cd1d77c7bf9f02c\": rpc error: code = NotFound desc = could not find container \"a8e3baecc27c1cd088d455a47b3231c90ef456655e1057400cd1d77c7bf9f02c\": container with ID starting with a8e3baecc27c1cd088d455a47b3231c90ef456655e1057400cd1d77c7bf9f02c not found: ID does not exist" Nov 02 21:38:00.024806 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:00.024767 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa406b90-6571-4f44-a9ba-b3299863e736-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:00.024806 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:00.024803 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7vnl\" (UniqueName: \"kubernetes.io/projected/aa406b90-6571-4f44-a9ba-b3299863e736-kube-api-access-t7vnl\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:00.025024 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:00.024818 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa406b90-6571-4f44-a9ba-b3299863e736-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:00.025024 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:00.024832 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa406b90-6571-4f44-a9ba-b3299863e736-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:00.141396 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:00.141309 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq"] Nov 02 21:38:00.144838 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:00.144811 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-682116-76bfcdc6b7-mmfxq"] Nov 02 21:38:01.438954 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:01.438919 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:01.779031 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:01.778997 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa406b90-6571-4f44-a9ba-b3299863e736" path="/var/lib/kubelet/pods/aa406b90-6571-4f44-a9ba-b3299863e736/volumes" Nov 02 21:38:04.356586 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:04.356539 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:04.581191 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:04.581145 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:05.943941 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:05.943906 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:06.468751 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:06.468714 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:06.683020 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:06.682982 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:07.189409 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:07.189374 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:12.264540 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:12.264508 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:12.877043 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:12.877007 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:14.210428 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:14.210392 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:14.413214 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:14.413167 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:15.933873 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:15.933839 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:16.448335 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:16.448291 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:16.693600 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:16.693553 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:17.386504 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:17.386470 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf"] Nov 02 21:38:17.387036 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:17.387012 4935 cpu_manager.go:414] "containerMap: removing container" podUID="4efcdacc-c172-4f9a-b877-3b0a510252de" containerName="payload-pre-processing" Nov 02 21:38:17.387036 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:17.387027 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4efcdacc-c172-4f9a-b877-3b0a510252de" containerName="payload-pre-processing" Nov 02 21:38:17.387119 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:17.387051 4935 cpu_manager.go:414] "containerMap: removing container" podUID="f58a753c-53b9-4570-8875-6123f00fed56" containerName="main" Nov 02 21:38:17.387119 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:17.387057 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58a753c-53b9-4570-8875-6123f00fed56" containerName="main" Nov 02 21:38:17.387119 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:17.387080 4935 cpu_manager.go:414] "containerMap: removing container" podUID="aa406b90-6571-4f44-a9ba-b3299863e736" containerName="maas-api" Nov 02 21:38:17.387119 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:17.387085 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa406b90-6571-4f44-a9ba-b3299863e736" containerName="maas-api" Nov 02 21:38:17.387285 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:17.387238 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="4efcdacc-c172-4f9a-b877-3b0a510252de" containerName="payload-pre-processing" Nov 02 21:38:17.387285 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:17.387261 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58a753c-53b9-4570-8875-6123f00fed56" containerName="main" Nov 02 21:38:17.387285 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:17.387273 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa406b90-6571-4f44-a9ba-b3299863e736" containerName="maas-api" Nov 02 21:38:17.392220 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:17.392199 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf" Nov 02 21:38:17.511192 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:17.511114 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf"] Nov 02 21:38:17.549334 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:17.548563 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-7d3617-serving-cert\"" Nov 02 21:38:17.566147 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:17.566092 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6616a037-8eba-4c91-9a4e-a1980773f2b1-metrics-certs\") pod \"maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf\" (UID: \"6616a037-8eba-4c91-9a4e-a1980773f2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf" Nov 02 21:38:17.566540 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:17.566514 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6616a037-8eba-4c91-9a4e-a1980773f2b1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf\" (UID: \"6616a037-8eba-4c91-9a4e-a1980773f2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf" Nov 02 21:38:17.566775 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:17.566761 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6616a037-8eba-4c91-9a4e-a1980773f2b1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf\" (UID: \"6616a037-8eba-4c91-9a4e-a1980773f2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf" Nov 02 21:38:17.566899 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:17.566884 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4plzt\" (UniqueName: \"kubernetes.io/projected/6616a037-8eba-4c91-9a4e-a1980773f2b1-kube-api-access-4plzt\") pod \"maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf\" (UID: \"6616a037-8eba-4c91-9a4e-a1980773f2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf" Nov 02 21:38:17.664707 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:17.664598 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:17.667949 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:17.667920 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6616a037-8eba-4c91-9a4e-a1980773f2b1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf\" (UID: \"6616a037-8eba-4c91-9a4e-a1980773f2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf" Nov 02 21:38:17.668115 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:17.667962 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4plzt\" (UniqueName: \"kubernetes.io/projected/6616a037-8eba-4c91-9a4e-a1980773f2b1-kube-api-access-4plzt\") pod \"maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf\" (UID: \"6616a037-8eba-4c91-9a4e-a1980773f2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf" Nov 02 21:38:17.668115 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:17.668001 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6616a037-8eba-4c91-9a4e-a1980773f2b1-metrics-certs\") pod \"maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf\" (UID: \"6616a037-8eba-4c91-9a4e-a1980773f2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf" Nov 02 21:38:17.668115 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:17.668106 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6616a037-8eba-4c91-9a4e-a1980773f2b1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf\" (UID: \"6616a037-8eba-4c91-9a4e-a1980773f2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf" Nov 02 21:38:17.668892 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:17.668867 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6616a037-8eba-4c91-9a4e-a1980773f2b1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf\" (UID: \"6616a037-8eba-4c91-9a4e-a1980773f2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf" Nov 02 21:38:17.671050 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:17.671025 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6616a037-8eba-4c91-9a4e-a1980773f2b1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf\" (UID: \"6616a037-8eba-4c91-9a4e-a1980773f2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf" Nov 02 21:38:17.671268 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:17.671243 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6616a037-8eba-4c91-9a4e-a1980773f2b1-metrics-certs\") pod \"maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf\" (UID: \"6616a037-8eba-4c91-9a4e-a1980773f2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf" Nov 02 21:38:17.724398 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:17.724365 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4plzt\" (UniqueName: \"kubernetes.io/projected/6616a037-8eba-4c91-9a4e-a1980773f2b1-kube-api-access-4plzt\") pod \"maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf\" (UID: \"6616a037-8eba-4c91-9a4e-a1980773f2b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf" Nov 02 21:38:17.907693 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:17.907647 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:18.005963 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:18.005872 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf" Nov 02 21:38:18.120507 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:18.120469 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:18.323563 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:18.323499 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:18.390697 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:18.390666 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf"] Nov 02 21:38:18.391936 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:38:18.391901 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6616a037_8eba_4c91_9a4e_a1980773f2b1.slice/crio-6eab084bf66915806c97f7efb8101963afe4cac249951c8110b33033805b0007 WatchSource:0}: Error finding container 6eab084bf66915806c97f7efb8101963afe4cac249951c8110b33033805b0007: Status 404 returned error can't find the container with id 6eab084bf66915806c97f7efb8101963afe4cac249951c8110b33033805b0007 Nov 02 21:38:18.855167 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:18.852747 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:19.087637 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:19.087605 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:19.122951 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:19.122872 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf" event={"ID":"6616a037-8eba-4c91-9a4e-a1980773f2b1","Type":"ContainerStarted","Data":"6eab084bf66915806c97f7efb8101963afe4cac249951c8110b33033805b0007"} Nov 02 21:38:19.291832 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:19.291804 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:20.160905 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:20.160858 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf" event={"ID":"6616a037-8eba-4c91-9a4e-a1980773f2b1","Type":"ContainerStarted","Data":"295f2fdcecbebab521d249856b5e049b66cb8aa8997b6b4e6b024ebf0c949ece"} Nov 02 21:38:20.162199 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:20.162173 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf" Nov 02 21:38:20.274422 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:20.274364 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf" podStartSLOduration=2.395891839 podStartE2EDuration="3.274348075s" totalImagesPullingTime="878.456236ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:38:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:38:20.274167749 +0000 UTC m=+1792.713742473" watchObservedRunningTime="2026-11-02 21:38:20.274348075 +0000 UTC m=+1792.713922798" Nov 02 21:38:22.923216 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:22.923172 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:24.507274 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:24.507225 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:24.933579 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:24.932245 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:25.468156 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:25.466548 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:25.697849 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:25.697800 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:26.546915 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:26.546871 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g"] Nov 02 21:38:26.547172 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:26.547110 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" podUID="00c2f98e-d274-4560-8f69-ca6e2dd1888d" containerName="istio-proxy" containerID="cri-o://66a939f5b5b849eb65ada94d6bc78b5c828e69ba5e0509c8bed9966f01bd24c8" gracePeriod=30 Nov 02 21:38:27.052809 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:27.052767 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:27.184121 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:27.184079 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf" Nov 02 21:38:27.505588 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:27.505498 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:27.706311 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:27.706275 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:28.135440 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:28.135408 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 21:38:28.144996 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:28.144969 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 21:38:28.144996 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:28.144986 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 21:38:28.153408 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:28.153388 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 21:38:28.420563 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:28.420483 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:28.470016 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:38:28.469982 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:38:29.128196 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:29.128124 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:29.352188 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:29.352114 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:30.529957 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:30.529918 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:31.822124 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:31.822070 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:38:32.010280 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.010241 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istiod-ca-cert\") pod \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " Nov 02 21:38:32.010471 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.010300 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istio-token\" (UniqueName: \"kubernetes.io/projected/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istio-token\") pod \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " Nov 02 21:38:32.010471 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.010348 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-credential-socket\") pod \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " Nov 02 21:38:32.010471 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.010429 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-workload-certs\") pod \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " Nov 02 21:38:32.010646 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.010573 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istio-envoy\") pod \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " Nov 02 21:38:32.010646 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.010603 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-workload-socket\") pod \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " Nov 02 21:38:32.010747 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.010655 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c2f98e-d274-4560-8f69-ca6e2dd1888d-kube-api-access-wjxgx\" (UniqueName: \"kubernetes.io/projected/00c2f98e-d274-4560-8f69-ca6e2dd1888d-kube-api-access-wjxgx\") pod \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " Nov 02 21:38:32.010747 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.010692 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istio-data\") pod \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " Nov 02 21:38:32.010747 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.010703 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istiod-ca-cert" pod "00c2f98e-d274-4560-8f69-ca6e2dd1888d" (UID: "00c2f98e-d274-4560-8f69-ca6e2dd1888d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:38:32.010898 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.010766 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istio-podinfo\") pod \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\" (UID: \"00c2f98e-d274-4560-8f69-ca6e2dd1888d\") " Nov 02 21:38:32.011215 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.011188 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-workload-certs" pod "00c2f98e-d274-4560-8f69-ca6e2dd1888d" (UID: "00c2f98e-d274-4560-8f69-ca6e2dd1888d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:32.011320 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.011200 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:32.011320 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.011281 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istio-data" pod "00c2f98e-d274-4560-8f69-ca6e2dd1888d" (UID: "00c2f98e-d274-4560-8f69-ca6e2dd1888d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:32.011478 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.011455 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-workload-socket" pod "00c2f98e-d274-4560-8f69-ca6e2dd1888d" (UID: "00c2f98e-d274-4560-8f69-ca6e2dd1888d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:32.011808 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.011707 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-credential-socket" pod "00c2f98e-d274-4560-8f69-ca6e2dd1888d" (UID: "00c2f98e-d274-4560-8f69-ca6e2dd1888d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:32.013916 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.013885 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istio-podinfo" pod "00c2f98e-d274-4560-8f69-ca6e2dd1888d" (UID: "00c2f98e-d274-4560-8f69-ca6e2dd1888d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:38:32.014009 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.013923 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c2f98e-d274-4560-8f69-ca6e2dd1888d-kube-api-access-wjxgx" pod "00c2f98e-d274-4560-8f69-ca6e2dd1888d" (UID: "00c2f98e-d274-4560-8f69-ca6e2dd1888d"). InnerVolumeSpecName "kube-api-access-wjxgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:38:32.014009 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.013958 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istio-token" pod "00c2f98e-d274-4560-8f69-ca6e2dd1888d" (UID: "00c2f98e-d274-4560-8f69-ca6e2dd1888d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:38:32.014435 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.014414 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istio-envoy" pod "00c2f98e-d274-4560-8f69-ca6e2dd1888d" (UID: "00c2f98e-d274-4560-8f69-ca6e2dd1888d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:32.112453 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.112376 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:32.112453 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.112404 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:32.112453 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.112414 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjxgx\" (UniqueName: \"kubernetes.io/projected/00c2f98e-d274-4560-8f69-ca6e2dd1888d-kube-api-access-wjxgx\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:32.112453 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.112424 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:32.112453 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.112433 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:32.112453 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.112441 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00c2f98e-d274-4560-8f69-ca6e2dd1888d-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:32.112453 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.112449 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:32.112453 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.112457 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c2f98e-d274-4560-8f69-ca6e2dd1888d-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:32.267393 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.267351 4935 generic.go:356] "Generic (PLEG): container finished" podID="00c2f98e-d274-4560-8f69-ca6e2dd1888d" containerID="66a939f5b5b849eb65ada94d6bc78b5c828e69ba5e0509c8bed9966f01bd24c8" exitCode=0 Nov 02 21:38:32.267591 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.267406 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" event={"ID":"00c2f98e-d274-4560-8f69-ca6e2dd1888d","Type":"ContainerDied","Data":"66a939f5b5b849eb65ada94d6bc78b5c828e69ba5e0509c8bed9966f01bd24c8"} Nov 02 21:38:32.267591 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.267412 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" Nov 02 21:38:32.267591 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.267460 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g" event={"ID":"00c2f98e-d274-4560-8f69-ca6e2dd1888d","Type":"ContainerDied","Data":"4f8f630541fc13cf4a1987215d11fd5cbd9a08c087e1a7b074aa1b9fd9f1c2b5"} Nov 02 21:38:32.267591 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.267491 4935 scope.go:122] "RemoveContainer" containerID="66a939f5b5b849eb65ada94d6bc78b5c828e69ba5e0509c8bed9966f01bd24c8" Nov 02 21:38:32.281361 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.281315 4935 scope.go:122] "RemoveContainer" containerID="66a939f5b5b849eb65ada94d6bc78b5c828e69ba5e0509c8bed9966f01bd24c8" Nov 02 21:38:32.281796 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:38:32.281772 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66a939f5b5b849eb65ada94d6bc78b5c828e69ba5e0509c8bed9966f01bd24c8\": container with ID starting with 66a939f5b5b849eb65ada94d6bc78b5c828e69ba5e0509c8bed9966f01bd24c8 not found: ID does not exist" containerID="66a939f5b5b849eb65ada94d6bc78b5c828e69ba5e0509c8bed9966f01bd24c8" Nov 02 21:38:32.281868 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.281808 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66a939f5b5b849eb65ada94d6bc78b5c828e69ba5e0509c8bed9966f01bd24c8"} err="failed to get container status \"66a939f5b5b849eb65ada94d6bc78b5c828e69ba5e0509c8bed9966f01bd24c8\": rpc error: code = NotFound desc = could not find container \"66a939f5b5b849eb65ada94d6bc78b5c828e69ba5e0509c8bed9966f01bd24c8\": container with ID starting with 66a939f5b5b849eb65ada94d6bc78b5c828e69ba5e0509c8bed9966f01bd24c8 not found: ID does not exist" Nov 02 21:38:32.422162 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.422036 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g"] Nov 02 21:38:32.425316 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:32.425290 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-qnz6g"] Nov 02 21:38:33.776983 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:33.776944 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00c2f98e-d274-4560-8f69-ca6e2dd1888d" path="/var/lib/kubelet/pods/00c2f98e-d274-4560-8f69-ca6e2dd1888d/volumes" Nov 02 21:38:34.032235 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:34.032120 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:34.266794 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:34.266741 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:35.963246 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:35.963207 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:37.206230 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:37.206193 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:37.409695 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:37.409658 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:37.924446 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:37.924408 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:38.124751 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:38.124703 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:41.177808 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:41.177775 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:42.549042 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:42.548979 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:43.057431 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:43.057395 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:43.262434 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:43.262395 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:46.700285 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:46.700238 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:51.942910 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:51.942815 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:56.961888 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:38:56.961840 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:38:58.471527 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:38:58.471487 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:39:02.236736 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:02.236693 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:05.866361 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:05.866323 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:06.506797 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:06.506751 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:09.454739 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:09.454666 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:14.456868 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:14.456832 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:15.821143 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:15.821106 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:16.346991 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:16.346952 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:16.558112 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:16.558075 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:19.991330 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:19.990058 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:23.660415 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:23.660376 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:24.302155 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:24.301072 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:24.587301 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:24.587220 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:25.261844 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:25.261782 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:25.462470 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:25.462424 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:25.662021 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:25.661979 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:27.824730 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:27.824692 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:28.471092 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:39:28.471055 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:39:28.547408 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:28.547322 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:28.768621 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:28.768586 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:31.173960 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:31.173915 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:33.914308 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:33.914268 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:34.114000 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:34.113962 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:35.215528 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:35.215497 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:35.739778 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:35.739732 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:38.841076 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:38.841036 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:43.913666 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:43.913632 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:49.035806 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:49.035759 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:54.112088 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:54.112042 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:55.586912 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:55.586874 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:56.197456 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:56.197064 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:56.492296 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:56.492198 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:39:58.475338 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:39:58.475293 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:39:59.377767 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:39:59.377729 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:00.749310 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:00.749263 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:00.970414 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:00.970377 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:04.266718 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:04.266685 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:04.472654 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:04.472614 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:06.167099 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:06.167057 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:07.982667 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:07.982631 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:08.283834 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:08.283797 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:08.363430 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:08.363388 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46"] Nov 02 21:40:08.363697 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:08.363667 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" podUID="8e2fddf5-ee5d-4872-91c2-b073e5261728" containerName="istio-proxy" containerID="cri-o://057ae25402254f08f401e9385b4930c01105fd64974dca249305f299afb43967" gracePeriod=30 Nov 02 21:40:08.373521 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:08.373490 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb"] Nov 02 21:40:08.373752 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:08.373727 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" podUID="3ea341f9-33df-461e-9f33-69be35dea3ba" containerName="istio-proxy" containerID="cri-o://5ee6104d02815b02ef1af01db00f3317d180091cddd1a27e44044f06f1eef0ae" gracePeriod=30 Nov 02 21:40:08.501741 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:08.501702 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:08.582428 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:08.582337 4935 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:40:08.582621 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:08.582424 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" podUID="3ea341f9-33df-461e-9f33-69be35dea3ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:40:08.771046 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:08.771008 4935 patch_prober.go:29] interesting pod/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:40:08.771224 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:08.771085 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" podUID="8e2fddf5-ee5d-4872-91c2-b073e5261728" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:40:12.174537 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:12.174487 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:12.717240 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:12.717198 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:13.672163 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.672113 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:40:13.719743 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.719719 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:40:13.780834 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.780804 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-credential-socket\") pod \"8e2fddf5-ee5d-4872-91c2-b073e5261728\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " Nov 02 21:40:13.780834 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.780834 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-istio-envoy\") pod \"3ea341f9-33df-461e-9f33-69be35dea3ba\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " Nov 02 21:40:13.781056 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.780890 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-credential-socket\") pod \"3ea341f9-33df-461e-9f33-69be35dea3ba\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " Nov 02 21:40:13.781056 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.780922 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-workload-socket\") pod \"8e2fddf5-ee5d-4872-91c2-b073e5261728\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " Nov 02 21:40:13.781056 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.780949 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-workload-certs\") pod \"3ea341f9-33df-461e-9f33-69be35dea3ba\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " Nov 02 21:40:13.781056 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.780979 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ea341f9-33df-461e-9f33-69be35dea3ba-istio-token\" (UniqueName: \"kubernetes.io/projected/3ea341f9-33df-461e-9f33-69be35dea3ba-istio-token\") pod \"3ea341f9-33df-461e-9f33-69be35dea3ba\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " Nov 02 21:40:13.781056 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.781013 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ea341f9-33df-461e-9f33-69be35dea3ba-kube-api-access-92x2k\" (UniqueName: \"kubernetes.io/projected/3ea341f9-33df-461e-9f33-69be35dea3ba-kube-api-access-92x2k\") pod \"3ea341f9-33df-461e-9f33-69be35dea3ba\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " Nov 02 21:40:13.781342 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.781056 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-istio-data\") pod \"8e2fddf5-ee5d-4872-91c2-b073e5261728\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " Nov 02 21:40:13.781342 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.781092 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-credential-socket" pod "8e2fddf5-ee5d-4872-91c2-b073e5261728" (UID: "8e2fddf5-ee5d-4872-91c2-b073e5261728"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:13.781342 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.781101 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-istio-envoy\") pod \"8e2fddf5-ee5d-4872-91c2-b073e5261728\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " Nov 02 21:40:13.781342 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.781171 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-workload-socket\") pod \"3ea341f9-33df-461e-9f33-69be35dea3ba\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " Nov 02 21:40:13.781342 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.781243 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-workload-certs" pod "3ea341f9-33df-461e-9f33-69be35dea3ba" (UID: "3ea341f9-33df-461e-9f33-69be35dea3ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:13.781342 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.781257 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-workload-socket" pod "8e2fddf5-ee5d-4872-91c2-b073e5261728" (UID: "8e2fddf5-ee5d-4872-91c2-b073e5261728"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:13.781342 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.781267 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-credential-socket" pod "3ea341f9-33df-461e-9f33-69be35dea3ba" (UID: "3ea341f9-33df-461e-9f33-69be35dea3ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:13.781342 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.781285 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ea341f9-33df-461e-9f33-69be35dea3ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ea341f9-33df-461e-9f33-69be35dea3ba-istiod-ca-cert\") pod \"3ea341f9-33df-461e-9f33-69be35dea3ba\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " Nov 02 21:40:13.781342 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.781343 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e2fddf5-ee5d-4872-91c2-b073e5261728-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2fddf5-ee5d-4872-91c2-b073e5261728-istio-podinfo\") pod \"8e2fddf5-ee5d-4872-91c2-b073e5261728\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " Nov 02 21:40:13.781802 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.781377 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ea341f9-33df-461e-9f33-69be35dea3ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ea341f9-33df-461e-9f33-69be35dea3ba-istio-podinfo\") pod \"3ea341f9-33df-461e-9f33-69be35dea3ba\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " Nov 02 21:40:13.781802 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.781416 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-workload-socket" pod "3ea341f9-33df-461e-9f33-69be35dea3ba" (UID: "3ea341f9-33df-461e-9f33-69be35dea3ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:13.781802 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.781453 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-istio-data" pod "8e2fddf5-ee5d-4872-91c2-b073e5261728" (UID: "8e2fddf5-ee5d-4872-91c2-b073e5261728"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:13.781802 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.781463 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e2fddf5-ee5d-4872-91c2-b073e5261728-kube-api-access-9hkpn\" (UniqueName: \"kubernetes.io/projected/8e2fddf5-ee5d-4872-91c2-b073e5261728-kube-api-access-9hkpn\") pod \"8e2fddf5-ee5d-4872-91c2-b073e5261728\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " Nov 02 21:40:13.781802 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.781562 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e2fddf5-ee5d-4872-91c2-b073e5261728-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2fddf5-ee5d-4872-91c2-b073e5261728-istiod-ca-cert\") pod \"8e2fddf5-ee5d-4872-91c2-b073e5261728\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " Nov 02 21:40:13.781802 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.781648 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-workload-certs\") pod \"8e2fddf5-ee5d-4872-91c2-b073e5261728\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " Nov 02 21:40:13.781802 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.781675 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-istio-data\") pod \"3ea341f9-33df-461e-9f33-69be35dea3ba\" (UID: \"3ea341f9-33df-461e-9f33-69be35dea3ba\") " Nov 02 21:40:13.781802 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.781718 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e2fddf5-ee5d-4872-91c2-b073e5261728-istio-token\" (UniqueName: \"kubernetes.io/projected/8e2fddf5-ee5d-4872-91c2-b073e5261728-istio-token\") pod \"8e2fddf5-ee5d-4872-91c2-b073e5261728\" (UID: \"8e2fddf5-ee5d-4872-91c2-b073e5261728\") " Nov 02 21:40:13.782220 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.781675 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea341f9-33df-461e-9f33-69be35dea3ba-istiod-ca-cert" pod "3ea341f9-33df-461e-9f33-69be35dea3ba" (UID: "3ea341f9-33df-461e-9f33-69be35dea3ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:40:13.782284 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.782215 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-istio-data" pod "3ea341f9-33df-461e-9f33-69be35dea3ba" (UID: "3ea341f9-33df-461e-9f33-69be35dea3ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:13.782284 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.782270 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.782397 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.782290 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.782397 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.782303 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.782397 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.782258 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-workload-certs" pod "8e2fddf5-ee5d-4872-91c2-b073e5261728" (UID: "8e2fddf5-ee5d-4872-91c2-b073e5261728"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:13.782397 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.782316 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.782397 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.782351 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.782397 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.782372 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.782731 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.782663 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e2fddf5-ee5d-4872-91c2-b073e5261728-istiod-ca-cert" pod "8e2fddf5-ee5d-4872-91c2-b073e5261728" (UID: "8e2fddf5-ee5d-4872-91c2-b073e5261728"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:40:13.783757 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.783722 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-istio-envoy" pod "3ea341f9-33df-461e-9f33-69be35dea3ba" (UID: "3ea341f9-33df-461e-9f33-69be35dea3ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:13.783862 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.783756 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-istio-envoy" pod "8e2fddf5-ee5d-4872-91c2-b073e5261728" (UID: "8e2fddf5-ee5d-4872-91c2-b073e5261728"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:13.783928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.783877 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2fddf5-ee5d-4872-91c2-b073e5261728-kube-api-access-9hkpn" pod "8e2fddf5-ee5d-4872-91c2-b073e5261728" (UID: "8e2fddf5-ee5d-4872-91c2-b073e5261728"). InnerVolumeSpecName "kube-api-access-9hkpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:40:13.783990 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.783964 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea341f9-33df-461e-9f33-69be35dea3ba-istio-token" pod "3ea341f9-33df-461e-9f33-69be35dea3ba" (UID: "3ea341f9-33df-461e-9f33-69be35dea3ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:40:13.784349 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.784330 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e2fddf5-ee5d-4872-91c2-b073e5261728-istio-podinfo" pod "8e2fddf5-ee5d-4872-91c2-b073e5261728" (UID: "8e2fddf5-ee5d-4872-91c2-b073e5261728"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:40:13.784683 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.784666 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea341f9-33df-461e-9f33-69be35dea3ba-kube-api-access-92x2k" pod "3ea341f9-33df-461e-9f33-69be35dea3ba" (UID: "3ea341f9-33df-461e-9f33-69be35dea3ba"). InnerVolumeSpecName "kube-api-access-92x2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:40:13.784822 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.784807 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ea341f9-33df-461e-9f33-69be35dea3ba-istio-podinfo" pod "3ea341f9-33df-461e-9f33-69be35dea3ba" (UID: "3ea341f9-33df-461e-9f33-69be35dea3ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:40:13.785203 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.785185 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2fddf5-ee5d-4872-91c2-b073e5261728-istio-token" pod "8e2fddf5-ee5d-4872-91c2-b073e5261728" (UID: "8e2fddf5-ee5d-4872-91c2-b073e5261728"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:40:13.886542 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.886444 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.886542 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.886485 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ea341f9-33df-461e-9f33-69be35dea3ba-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.886542 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.886514 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2fddf5-ee5d-4872-91c2-b073e5261728-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.886542 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.886528 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ea341f9-33df-461e-9f33-69be35dea3ba-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.886876 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.886574 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hkpn\" (UniqueName: \"kubernetes.io/projected/8e2fddf5-ee5d-4872-91c2-b073e5261728-kube-api-access-9hkpn\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.886876 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.886588 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2fddf5-ee5d-4872-91c2-b073e5261728-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.886876 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.886607 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2fddf5-ee5d-4872-91c2-b073e5261728-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.886876 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.886622 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.886876 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.886635 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2fddf5-ee5d-4872-91c2-b073e5261728-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.886876 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.886648 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ea341f9-33df-461e-9f33-69be35dea3ba-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.886876 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.886664 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ea341f9-33df-461e-9f33-69be35dea3ba-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.886876 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.886677 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92x2k\" (UniqueName: \"kubernetes.io/projected/3ea341f9-33df-461e-9f33-69be35dea3ba-kube-api-access-92x2k\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:13.994259 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.994218 4935 generic.go:356] "Generic (PLEG): container finished" podID="8e2fddf5-ee5d-4872-91c2-b073e5261728" containerID="057ae25402254f08f401e9385b4930c01105fd64974dca249305f299afb43967" exitCode=0 Nov 02 21:40:13.994463 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.994300 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" Nov 02 21:40:13.994463 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.994306 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" event={"ID":"8e2fddf5-ee5d-4872-91c2-b073e5261728","Type":"ContainerDied","Data":"057ae25402254f08f401e9385b4930c01105fd64974dca249305f299afb43967"} Nov 02 21:40:13.994463 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.994437 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46" event={"ID":"8e2fddf5-ee5d-4872-91c2-b073e5261728","Type":"ContainerDied","Data":"3b925e1f9f30d0d54621cb580c9991362e3d4455dc82959118f7f297df486079"} Nov 02 21:40:13.994657 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.994475 4935 scope.go:122] "RemoveContainer" containerID="057ae25402254f08f401e9385b4930c01105fd64974dca249305f299afb43967" Nov 02 21:40:13.995977 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.995953 4935 generic.go:356] "Generic (PLEG): container finished" podID="3ea341f9-33df-461e-9f33-69be35dea3ba" containerID="5ee6104d02815b02ef1af01db00f3317d180091cddd1a27e44044f06f1eef0ae" exitCode=0 Nov 02 21:40:13.996072 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.996014 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" Nov 02 21:40:13.996072 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.996030 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" event={"ID":"3ea341f9-33df-461e-9f33-69be35dea3ba","Type":"ContainerDied","Data":"5ee6104d02815b02ef1af01db00f3317d180091cddd1a27e44044f06f1eef0ae"} Nov 02 21:40:13.996199 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:13.996087 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb" event={"ID":"3ea341f9-33df-461e-9f33-69be35dea3ba","Type":"ContainerDied","Data":"d2415e5d5e3b190fd2854ae22f823e6a133002fee180e7a3f10fd1bdc70b48f8"} Nov 02 21:40:14.006744 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:14.006717 4935 scope.go:122] "RemoveContainer" containerID="057ae25402254f08f401e9385b4930c01105fd64974dca249305f299afb43967" Nov 02 21:40:14.007040 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:40:14.007022 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"057ae25402254f08f401e9385b4930c01105fd64974dca249305f299afb43967\": container with ID starting with 057ae25402254f08f401e9385b4930c01105fd64974dca249305f299afb43967 not found: ID does not exist" containerID="057ae25402254f08f401e9385b4930c01105fd64974dca249305f299afb43967" Nov 02 21:40:14.007105 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:14.007050 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"057ae25402254f08f401e9385b4930c01105fd64974dca249305f299afb43967"} err="failed to get container status \"057ae25402254f08f401e9385b4930c01105fd64974dca249305f299afb43967\": rpc error: code = NotFound desc = could not find container \"057ae25402254f08f401e9385b4930c01105fd64974dca249305f299afb43967\": container with ID starting with 057ae25402254f08f401e9385b4930c01105fd64974dca249305f299afb43967 not found: ID does not exist" Nov 02 21:40:14.007105 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:14.007073 4935 scope.go:122] "RemoveContainer" containerID="5ee6104d02815b02ef1af01db00f3317d180091cddd1a27e44044f06f1eef0ae" Nov 02 21:40:14.016166 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:14.016118 4935 scope.go:122] "RemoveContainer" containerID="5ee6104d02815b02ef1af01db00f3317d180091cddd1a27e44044f06f1eef0ae" Nov 02 21:40:14.016439 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:40:14.016419 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ee6104d02815b02ef1af01db00f3317d180091cddd1a27e44044f06f1eef0ae\": container with ID starting with 5ee6104d02815b02ef1af01db00f3317d180091cddd1a27e44044f06f1eef0ae not found: ID does not exist" containerID="5ee6104d02815b02ef1af01db00f3317d180091cddd1a27e44044f06f1eef0ae" Nov 02 21:40:14.016510 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:14.016448 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ee6104d02815b02ef1af01db00f3317d180091cddd1a27e44044f06f1eef0ae"} err="failed to get container status \"5ee6104d02815b02ef1af01db00f3317d180091cddd1a27e44044f06f1eef0ae\": rpc error: code = NotFound desc = could not find container \"5ee6104d02815b02ef1af01db00f3317d180091cddd1a27e44044f06f1eef0ae\": container with ID starting with 5ee6104d02815b02ef1af01db00f3317d180091cddd1a27e44044f06f1eef0ae not found: ID does not exist" Nov 02 21:40:14.149879 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:14.149789 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb"] Nov 02 21:40:14.154060 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:14.154031 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-mqnvb"] Nov 02 21:40:14.254797 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:14.254758 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46"] Nov 02 21:40:14.258744 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:14.258704 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-70efbbab-openshift-default-85b7d96449-tfn46"] Nov 02 21:40:15.163334 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:15.163285 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:15.392431 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:15.392396 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:15.717943 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:15.717898 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:15.777447 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:15.777417 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ea341f9-33df-461e-9f33-69be35dea3ba" path="/var/lib/kubelet/pods/3ea341f9-33df-461e-9f33-69be35dea3ba/volumes" Nov 02 21:40:15.777922 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:15.777906 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e2fddf5-ee5d-4872-91c2-b073e5261728" path="/var/lib/kubelet/pods/8e2fddf5-ee5d-4872-91c2-b073e5261728/volumes" Nov 02 21:40:20.824327 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:20.824241 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:26.034690 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:26.034649 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:26.575773 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:26.575739 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:27.514101 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:27.514062 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:27.718472 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:27.718438 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:28.159943 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:28.159909 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:28.470603 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:40:28.470502 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:40:28.763810 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:28.763777 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:29.374911 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:29.371893 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:29.774089 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:29.774048 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:30.889845 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:30.889804 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f"] Nov 02 21:40:30.892213 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:30.892185 4935 cpu_manager.go:414] "containerMap: removing container" podUID="00c2f98e-d274-4560-8f69-ca6e2dd1888d" containerName="istio-proxy" Nov 02 21:40:30.892213 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:30.892213 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c2f98e-d274-4560-8f69-ca6e2dd1888d" containerName="istio-proxy" Nov 02 21:40:30.892369 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:30.892251 4935 cpu_manager.go:414] "containerMap: removing container" podUID="3ea341f9-33df-461e-9f33-69be35dea3ba" containerName="istio-proxy" Nov 02 21:40:30.892369 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:30.892258 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ea341f9-33df-461e-9f33-69be35dea3ba" containerName="istio-proxy" Nov 02 21:40:30.892369 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:30.892281 4935 cpu_manager.go:414] "containerMap: removing container" podUID="8e2fddf5-ee5d-4872-91c2-b073e5261728" containerName="istio-proxy" Nov 02 21:40:30.892369 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:30.892288 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e2fddf5-ee5d-4872-91c2-b073e5261728" containerName="istio-proxy" Nov 02 21:40:30.892493 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:30.892415 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c2f98e-d274-4560-8f69-ca6e2dd1888d" containerName="istio-proxy" Nov 02 21:40:30.892493 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:30.892433 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ea341f9-33df-461e-9f33-69be35dea3ba" containerName="istio-proxy" Nov 02 21:40:30.892493 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:30.892445 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e2fddf5-ee5d-4872-91c2-b073e5261728" containerName="istio-proxy" Nov 02 21:40:30.898670 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:30.898630 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f" Nov 02 21:40:30.948032 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:30.948002 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f"] Nov 02 21:40:30.952092 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:30.952058 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6w94\" (UniqueName: \"kubernetes.io/projected/1ea9903b-325f-4281-92b9-e608ec727ea4-kube-api-access-z6w94\") pod \"maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f\" (UID: \"1ea9903b-325f-4281-92b9-e608ec727ea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f" Nov 02 21:40:30.952287 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:30.952219 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea9903b-325f-4281-92b9-e608ec727ea4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f\" (UID: \"1ea9903b-325f-4281-92b9-e608ec727ea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f" Nov 02 21:40:30.952287 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:30.952283 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ea9903b-325f-4281-92b9-e608ec727ea4-metrics-certs\") pod \"maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f\" (UID: \"1ea9903b-325f-4281-92b9-e608ec727ea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f" Nov 02 21:40:30.952417 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:30.952368 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ea9903b-325f-4281-92b9-e608ec727ea4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f\" (UID: \"1ea9903b-325f-4281-92b9-e608ec727ea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f" Nov 02 21:40:30.989850 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:30.989816 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-4453bd-serving-cert\"" Nov 02 21:40:31.053218 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:31.053181 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ea9903b-325f-4281-92b9-e608ec727ea4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f\" (UID: \"1ea9903b-325f-4281-92b9-e608ec727ea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f" Nov 02 21:40:31.053403 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:31.053286 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6w94\" (UniqueName: \"kubernetes.io/projected/1ea9903b-325f-4281-92b9-e608ec727ea4-kube-api-access-z6w94\") pod \"maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f\" (UID: \"1ea9903b-325f-4281-92b9-e608ec727ea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f" Nov 02 21:40:31.053403 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:40:31.053337 4935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-serving-cert: secret "maas-api-e2e-ipp-b-4453bd-serving-cert" not found Nov 02 21:40:31.053403 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:31.053372 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea9903b-325f-4281-92b9-e608ec727ea4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f\" (UID: \"1ea9903b-325f-4281-92b9-e608ec727ea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f" Nov 02 21:40:31.053403 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:40:31.053400 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ea9903b-325f-4281-92b9-e608ec727ea4-maas-api-tls podName:1ea9903b-325f-4281-92b9-e608ec727ea4 nodeName:}" failed. No retries permitted until 2026-11-02 21:40:31.553385409 +0000 UTC m=+1923.992960108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1ea9903b-325f-4281-92b9-e608ec727ea4-maas-api-tls") pod "maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f" (UID: "1ea9903b-325f-4281-92b9-e608ec727ea4") : secret "maas-api-e2e-ipp-b-4453bd-serving-cert" not found Nov 02 21:40:31.053606 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:31.053447 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ea9903b-325f-4281-92b9-e608ec727ea4-metrics-certs\") pod \"maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f\" (UID: \"1ea9903b-325f-4281-92b9-e608ec727ea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f" Nov 02 21:40:31.054457 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:31.054434 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea9903b-325f-4281-92b9-e608ec727ea4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f\" (UID: \"1ea9903b-325f-4281-92b9-e608ec727ea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f" Nov 02 21:40:31.056012 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:31.055989 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ea9903b-325f-4281-92b9-e608ec727ea4-metrics-certs\") pod \"maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f\" (UID: \"1ea9903b-325f-4281-92b9-e608ec727ea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f" Nov 02 21:40:31.107896 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:31.107858 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6w94\" (UniqueName: \"kubernetes.io/projected/1ea9903b-325f-4281-92b9-e608ec727ea4-kube-api-access-z6w94\") pod \"maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f\" (UID: \"1ea9903b-325f-4281-92b9-e608ec727ea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f" Nov 02 21:40:31.219055 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:31.218954 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:31.426333 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:31.426292 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:31.559676 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:31.559632 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ea9903b-325f-4281-92b9-e608ec727ea4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f\" (UID: \"1ea9903b-325f-4281-92b9-e608ec727ea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f" Nov 02 21:40:31.561992 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:31.561968 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ea9903b-325f-4281-92b9-e608ec727ea4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f\" (UID: \"1ea9903b-325f-4281-92b9-e608ec727ea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f" Nov 02 21:40:31.626069 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:31.626029 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:31.815258 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:31.815179 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f" Nov 02 21:40:32.185466 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:32.185427 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f"] Nov 02 21:40:32.187431 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:40:32.187402 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea9903b_325f_4281_92b9_e608ec727ea4.slice/crio-da9e04e65b450a79a31018776add68e27e41093cc3393c632016421e97a2f6e2 WatchSource:0}: Error finding container da9e04e65b450a79a31018776add68e27e41093cc3393c632016421e97a2f6e2: Status 404 returned error can't find the container with id da9e04e65b450a79a31018776add68e27e41093cc3393c632016421e97a2f6e2 Nov 02 21:40:32.492613 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:32.492523 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:32.718457 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:32.718420 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:33.194541 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:33.194499 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f" event={"ID":"1ea9903b-325f-4281-92b9-e608ec727ea4","Type":"ContainerStarted","Data":"ea0771ddde4ef4fcd5870c257f7097b9ddd81367749429a657fc2d9925775a5a"} Nov 02 21:40:33.194897 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:33.194555 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f" event={"ID":"1ea9903b-325f-4281-92b9-e608ec727ea4","Type":"ContainerStarted","Data":"da9e04e65b450a79a31018776add68e27e41093cc3393c632016421e97a2f6e2"} Nov 02 21:40:33.194897 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:33.194610 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f" Nov 02 21:40:33.328551 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:33.328494 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f" podStartSLOduration=2.407612757 podStartE2EDuration="3.328479368s" totalImagesPullingTime="920.866611ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:33.327181704 +0000 UTC m=+1925.766756430" watchObservedRunningTime="2026-11-02 21:40:33.328479368 +0000 UTC m=+1925.768054090" Nov 02 21:40:35.244794 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:35.244756 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:35.502443 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:35.502350 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:36.767906 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:36.767858 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:37.356860 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:37.356827 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:37.638636 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:37.638544 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:38.222734 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:38.222696 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:38.577920 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:38.577880 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:38.779984 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:38.779946 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:39.208605 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:39.208574 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f" Nov 02 21:40:43.108319 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:43.099864 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:43.298766 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:43.298731 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:43.673170 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:43.673117 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:43.907019 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:43.906985 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:44.845913 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:44.845878 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:45.903884 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:45.903844 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:46.113100 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:46.113066 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:46.627460 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:46.627392 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:46.841276 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:46.841242 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:50.139256 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:50.139215 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:55.001893 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:55.001834 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:55.205123 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:55.205089 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:55.329464 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:55.329425 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch"] Nov 02 21:40:55.333472 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:55.333449 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch" Nov 02 21:40:55.386184 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:55.386155 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch"] Nov 02 21:40:55.414820 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:55.414789 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slf8r\" (UniqueName: \"kubernetes.io/projected/1fb23512-885f-4a8d-9f2c-4e6819e3b8b4-kube-api-access-slf8r\") pod \"payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch\" (UID: \"1fb23512-885f-4a8d-9f2c-4e6819e3b8b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch" Nov 02 21:40:55.415023 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:55.415004 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fb23512-885f-4a8d-9f2c-4e6819e3b8b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch\" (UID: \"1fb23512-885f-4a8d-9f2c-4e6819e3b8b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch" Nov 02 21:40:55.430104 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:55.430074 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3e3997-dockercfg-kckmh\"" Nov 02 21:40:55.430337 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:55.430248 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3e3997\"" Nov 02 21:40:55.436195 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:55.436126 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4"] Nov 02 21:40:55.450211 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:55.443681 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4" Nov 02 21:40:55.482889 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:55.474823 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:40:55.491309 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:55.491270 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4"] Nov 02 21:40:55.516293 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:55.516250 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p44tm\" (UniqueName: \"kubernetes.io/projected/a6f83b79-d7fe-49b7-9a9e-036522f072ab-kube-api-access-p44tm\") pod \"payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4\" (UID: \"a6f83b79-d7fe-49b7-9a9e-036522f072ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4" Nov 02 21:40:55.516473 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:55.516388 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6f83b79-d7fe-49b7-9a9e-036522f072ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4\" (UID: \"a6f83b79-d7fe-49b7-9a9e-036522f072ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4" Nov 02 21:40:55.516531 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:55.516476 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fb23512-885f-4a8d-9f2c-4e6819e3b8b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch\" (UID: \"1fb23512-885f-4a8d-9f2c-4e6819e3b8b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch" Nov 02 21:40:55.517236 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:55.517192 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slf8r\" (UniqueName: \"kubernetes.io/projected/1fb23512-885f-4a8d-9f2c-4e6819e3b8b4-kube-api-access-slf8r\") pod \"payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch\" (UID: \"1fb23512-885f-4a8d-9f2c-4e6819e3b8b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch" Nov 02 21:40:55.519588 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:55.519556 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fb23512-885f-4a8d-9f2c-4e6819e3b8b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch\" (UID: \"1fb23512-885f-4a8d-9f2c-4e6819e3b8b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch" Nov 02 21:40:55.573125 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:55.573093 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slf8r\" (UniqueName: \"kubernetes.io/projected/1fb23512-885f-4a8d-9f2c-4e6819e3b8b4-kube-api-access-slf8r\") pod \"payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch\" (UID: \"1fb23512-885f-4a8d-9f2c-4e6819e3b8b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch" Nov 02 21:40:55.618456 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:55.618373 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p44tm\" (UniqueName: \"kubernetes.io/projected/a6f83b79-d7fe-49b7-9a9e-036522f072ab-kube-api-access-p44tm\") pod \"payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4\" (UID: \"a6f83b79-d7fe-49b7-9a9e-036522f072ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4" Nov 02 21:40:55.618634 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:55.618503 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6f83b79-d7fe-49b7-9a9e-036522f072ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4\" (UID: \"a6f83b79-d7fe-49b7-9a9e-036522f072ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4" Nov 02 21:40:55.619288 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:55.619266 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6f83b79-d7fe-49b7-9a9e-036522f072ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4\" (UID: \"a6f83b79-d7fe-49b7-9a9e-036522f072ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4" Nov 02 21:40:55.648252 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:55.648221 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch" Nov 02 21:40:55.672244 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:55.672186 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p44tm\" (UniqueName: \"kubernetes.io/projected/a6f83b79-d7fe-49b7-9a9e-036522f072ab-kube-api-access-p44tm\") pod \"payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4\" (UID: \"a6f83b79-d7fe-49b7-9a9e-036522f072ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4" Nov 02 21:40:55.790505 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:55.790473 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4" Nov 02 21:40:56.021517 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:56.021490 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch"] Nov 02 21:40:56.022755 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:40:56.022729 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb23512_885f_4a8d_9f2c_4e6819e3b8b4.slice/crio-86ae4eec8090a6d55fc848c8dbfee4afddcc731f819ef2da1ccae99b47c47d6a WatchSource:0}: Error finding container 86ae4eec8090a6d55fc848c8dbfee4afddcc731f819ef2da1ccae99b47c47d6a: Status 404 returned error can't find the container with id 86ae4eec8090a6d55fc848c8dbfee4afddcc731f819ef2da1ccae99b47c47d6a Nov 02 21:40:56.170415 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:56.170385 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4"] Nov 02 21:40:56.173207 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:40:56.173055 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f83b79_d7fe_49b7_9a9e_036522f072ab.slice/crio-63a7bac6a0005fe158d0fa381a23c0552be8217677b340dcc8cef50ff66fad45 WatchSource:0}: Error finding container 63a7bac6a0005fe158d0fa381a23c0552be8217677b340dcc8cef50ff66fad45: Status 404 returned error can't find the container with id 63a7bac6a0005fe158d0fa381a23c0552be8217677b340dcc8cef50ff66fad45 Nov 02 21:40:56.380913 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:56.380822 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4" event={"ID":"a6f83b79-d7fe-49b7-9a9e-036522f072ab","Type":"ContainerStarted","Data":"5b20c9f8c7250307919f988bde4df564d4c98d1a12eadecf326f9e50a8749a39"} Nov 02 21:40:56.380913 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:56.380883 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4" event={"ID":"a6f83b79-d7fe-49b7-9a9e-036522f072ab","Type":"ContainerStarted","Data":"63a7bac6a0005fe158d0fa381a23c0552be8217677b340dcc8cef50ff66fad45"} Nov 02 21:40:56.380913 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:56.380909 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4" Nov 02 21:40:56.382584 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:56.382548 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch" event={"ID":"1fb23512-885f-4a8d-9f2c-4e6819e3b8b4","Type":"ContainerStarted","Data":"3a610378faed6d413bc7641c4a6578cf4f479fa73a6b2e30a21f00f5672f2bfb"} Nov 02 21:40:56.382707 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:56.382595 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch" event={"ID":"1fb23512-885f-4a8d-9f2c-4e6819e3b8b4","Type":"ContainerStarted","Data":"86ae4eec8090a6d55fc848c8dbfee4afddcc731f819ef2da1ccae99b47c47d6a"} Nov 02 21:40:56.382707 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:56.382632 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch" Nov 02 21:40:56.489084 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:56.489029 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4" podStartSLOduration=1.489014035 podStartE2EDuration="1.489014035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:56.487051188 +0000 UTC m=+1948.926625912" watchObservedRunningTime="2026-11-02 21:40:56.489014035 +0000 UTC m=+1948.928588757" Nov 02 21:40:56.592807 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:40:56.592755 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch" podStartSLOduration=1.592739728 podStartE2EDuration="1.592739728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:56.592199838 +0000 UTC m=+1949.031774560" watchObservedRunningTime="2026-11-02 21:40:56.592739728 +0000 UTC m=+1949.032314451" Nov 02 21:40:58.469215 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:40:58.469171 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:41:00.310660 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:41:00.310620 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:05.184088 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:41:05.184035 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:05.391633 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:41:05.391602 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:05.591079 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:41:05.591039 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:07.391618 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:41:07.391590 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch" Nov 02 21:41:07.392107 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:41:07.391669 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4" Nov 02 21:41:10.630633 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:41:10.630595 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:17.539153 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:41:17.537367 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:24.243940 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:41:24.243902 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:25.519213 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:41:25.519174 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:28.469877 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:41:28.469836 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:41:28.530264 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:41:28.530224 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:29.211502 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:41:29.211467 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:34.410966 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:41:34.410929 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:38.019860 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:41:38.019794 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:38.519630 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:41:38.519588 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:41.301838 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:41:41.301799 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:41.532988 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:41:41.532950 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:44.831197 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:41:44.831157 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:45.051690 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:41:45.051654 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:46.546557 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:41:46.546510 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:47.291764 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:41:47.291728 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:51.466088 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:41:51.466024 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:51.693690 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:41:51.693650 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:51.912007 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:41:51.911957 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:56.966091 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:41:56.966051 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:57.313787 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:41:57.313751 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:41:58.469515 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:41:58.469471 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:41:59.153741 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:41:59.153700 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:00.097073 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:00.097041 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:00.297058 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:00.297020 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:03.991815 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:03.991775 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:04.236300 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:04.236257 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:04.463739 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:04.463704 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:05.259879 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:05.259846 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:09.498189 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:09.498054 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:14.640152 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:14.640089 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:18.117691 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:18.117652 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:18.347720 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:18.347687 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:19.638980 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:19.638941 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:24.698698 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:24.698662 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:26.692362 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:26.692327 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:28.471360 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:42:28.471319 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:42:28.851006 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:28.850967 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:29.683388 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:29.683347 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:33.903926 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:33.903882 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc"] Nov 02 21:42:33.904451 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:33.904116 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc" podUID="45871915-74c7-4bf7-a338-31f16c746f76" containerName="maas-api" containerID="cri-o://3adc7d6437961cdac22534e95601415d4f220bcf0172754a3084e9f7f155d82b" gracePeriod=30 Nov 02 21:42:34.101735 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:34.101657 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:34.109329 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:34.109188 4935 generic.go:356] "Generic (PLEG): container finished" podID="45871915-74c7-4bf7-a338-31f16c746f76" containerID="3adc7d6437961cdac22534e95601415d4f220bcf0172754a3084e9f7f155d82b" exitCode=0 Nov 02 21:42:34.109329 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:34.109222 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc" event={"ID":"45871915-74c7-4bf7-a338-31f16c746f76","Type":"ContainerDied","Data":"3adc7d6437961cdac22534e95601415d4f220bcf0172754a3084e9f7f155d82b"} Nov 02 21:42:34.235928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:34.235901 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc" Nov 02 21:42:34.359500 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:34.359452 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45871915-74c7-4bf7-a338-31f16c746f76-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45871915-74c7-4bf7-a338-31f16c746f76-openshift-service-ca\") pod \"45871915-74c7-4bf7-a338-31f16c746f76\" (UID: \"45871915-74c7-4bf7-a338-31f16c746f76\") " Nov 02 21:42:34.359701 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:34.359624 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45871915-74c7-4bf7-a338-31f16c746f76-metrics-certs\" (UniqueName: \"kubernetes.io/secret/45871915-74c7-4bf7-a338-31f16c746f76-metrics-certs\") pod \"45871915-74c7-4bf7-a338-31f16c746f76\" (UID: \"45871915-74c7-4bf7-a338-31f16c746f76\") " Nov 02 21:42:34.359701 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:34.359692 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45871915-74c7-4bf7-a338-31f16c746f76-kube-api-access-swpt7\" (UniqueName: \"kubernetes.io/projected/45871915-74c7-4bf7-a338-31f16c746f76-kube-api-access-swpt7\") pod \"45871915-74c7-4bf7-a338-31f16c746f76\" (UID: \"45871915-74c7-4bf7-a338-31f16c746f76\") " Nov 02 21:42:34.359829 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:34.359797 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45871915-74c7-4bf7-a338-31f16c746f76-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45871915-74c7-4bf7-a338-31f16c746f76-maas-api-tls\") pod \"45871915-74c7-4bf7-a338-31f16c746f76\" (UID: \"45871915-74c7-4bf7-a338-31f16c746f76\") " Nov 02 21:42:34.359926 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:34.359894 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45871915-74c7-4bf7-a338-31f16c746f76-openshift-service-ca" pod "45871915-74c7-4bf7-a338-31f16c746f76" (UID: "45871915-74c7-4bf7-a338-31f16c746f76"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:42:34.360435 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:34.360404 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45871915-74c7-4bf7-a338-31f16c746f76-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:42:34.362604 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:34.362568 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45871915-74c7-4bf7-a338-31f16c746f76-kube-api-access-swpt7" pod "45871915-74c7-4bf7-a338-31f16c746f76" (UID: "45871915-74c7-4bf7-a338-31f16c746f76"). InnerVolumeSpecName "kube-api-access-swpt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:42:34.362847 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:34.362817 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45871915-74c7-4bf7-a338-31f16c746f76-maas-api-tls" pod "45871915-74c7-4bf7-a338-31f16c746f76" (UID: "45871915-74c7-4bf7-a338-31f16c746f76"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:42:34.367278 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:34.367236 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45871915-74c7-4bf7-a338-31f16c746f76-metrics-certs" pod "45871915-74c7-4bf7-a338-31f16c746f76" (UID: "45871915-74c7-4bf7-a338-31f16c746f76"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:42:34.461723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:34.461637 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45871915-74c7-4bf7-a338-31f16c746f76-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:42:34.461723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:34.461671 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swpt7\" (UniqueName: \"kubernetes.io/projected/45871915-74c7-4bf7-a338-31f16c746f76-kube-api-access-swpt7\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:42:34.461723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:34.461681 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45871915-74c7-4bf7-a338-31f16c746f76-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:42:34.624589 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:34.624555 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:34.860424 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:34.860371 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:35.118224 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:35.118121 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc" Nov 02 21:42:35.118224 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:35.118162 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc" event={"ID":"45871915-74c7-4bf7-a338-31f16c746f76","Type":"ContainerDied","Data":"7cebeff73723d82ea1f45dfa70af91ae0c8157aaa67ccc3a21799669f6b3191f"} Nov 02 21:42:35.118712 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:35.118234 4935 scope.go:122] "RemoveContainer" containerID="3adc7d6437961cdac22534e95601415d4f220bcf0172754a3084e9f7f155d82b" Nov 02 21:42:35.272017 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:35.271961 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc"] Nov 02 21:42:35.318500 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:35.318466 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-54e844-886898c54-vb7gc"] Nov 02 21:42:35.786197 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:35.786162 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45871915-74c7-4bf7-a338-31f16c746f76" path="/var/lib/kubelet/pods/45871915-74c7-4bf7-a338-31f16c746f76/volumes" Nov 02 21:42:36.336447 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:36.336408 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:36.592853 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:36.592771 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:37.926543 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:37.926508 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:38.478505 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:38.478461 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:39.235047 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:39.235013 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:39.457479 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:39.457437 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:39.682231 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:39.682191 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:43.800025 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:43.799988 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:45.837839 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:45.837797 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:49.230159 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:49.229820 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:54.447832 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:42:54.447798 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:42:58.470568 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:42:58.470532 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:43:01.153257 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:01.153215 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:05.370215 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:05.370179 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f"] Nov 02 21:43:05.370678 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:05.370395 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f" podUID="1ea9903b-325f-4281-92b9-e608ec727ea4" containerName="maas-api" containerID="cri-o://ea0771ddde4ef4fcd5870c257f7097b9ddd81367749429a657fc2d9925775a5a" gracePeriod=30 Nov 02 21:43:05.555742 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:05.555703 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:05.738665 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:05.738636 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f" Nov 02 21:43:05.817464 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:05.816434 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ea9903b-325f-4281-92b9-e608ec727ea4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ea9903b-325f-4281-92b9-e608ec727ea4-maas-api-tls\") pod \"1ea9903b-325f-4281-92b9-e608ec727ea4\" (UID: \"1ea9903b-325f-4281-92b9-e608ec727ea4\") " Nov 02 21:43:05.817464 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:05.816512 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ea9903b-325f-4281-92b9-e608ec727ea4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea9903b-325f-4281-92b9-e608ec727ea4-openshift-service-ca\") pod \"1ea9903b-325f-4281-92b9-e608ec727ea4\" (UID: \"1ea9903b-325f-4281-92b9-e608ec727ea4\") " Nov 02 21:43:05.817464 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:05.816691 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ea9903b-325f-4281-92b9-e608ec727ea4-kube-api-access-z6w94\" (UniqueName: \"kubernetes.io/projected/1ea9903b-325f-4281-92b9-e608ec727ea4-kube-api-access-z6w94\") pod \"1ea9903b-325f-4281-92b9-e608ec727ea4\" (UID: \"1ea9903b-325f-4281-92b9-e608ec727ea4\") " Nov 02 21:43:05.817464 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:05.816736 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ea9903b-325f-4281-92b9-e608ec727ea4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ea9903b-325f-4281-92b9-e608ec727ea4-metrics-certs\") pod \"1ea9903b-325f-4281-92b9-e608ec727ea4\" (UID: \"1ea9903b-325f-4281-92b9-e608ec727ea4\") " Nov 02 21:43:05.819571 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:05.819529 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ea9903b-325f-4281-92b9-e608ec727ea4-openshift-service-ca" pod "1ea9903b-325f-4281-92b9-e608ec727ea4" (UID: "1ea9903b-325f-4281-92b9-e608ec727ea4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:43:05.822858 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:05.822829 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea9903b-325f-4281-92b9-e608ec727ea4-kube-api-access-z6w94" pod "1ea9903b-325f-4281-92b9-e608ec727ea4" (UID: "1ea9903b-325f-4281-92b9-e608ec727ea4"). InnerVolumeSpecName "kube-api-access-z6w94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:43:05.834355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:05.834299 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ea9903b-325f-4281-92b9-e608ec727ea4-metrics-certs" pod "1ea9903b-325f-4281-92b9-e608ec727ea4" (UID: "1ea9903b-325f-4281-92b9-e608ec727ea4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:43:05.834966 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:05.834908 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ea9903b-325f-4281-92b9-e608ec727ea4-maas-api-tls" pod "1ea9903b-325f-4281-92b9-e608ec727ea4" (UID: "1ea9903b-325f-4281-92b9-e608ec727ea4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:43:05.918384 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:05.918342 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ea9903b-325f-4281-92b9-e608ec727ea4-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:05.918384 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:05.918378 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea9903b-325f-4281-92b9-e608ec727ea4-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:05.918590 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:05.918391 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6w94\" (UniqueName: \"kubernetes.io/projected/1ea9903b-325f-4281-92b9-e608ec727ea4-kube-api-access-z6w94\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:05.918590 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:05.918406 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ea9903b-325f-4281-92b9-e608ec727ea4-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:06.335544 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:06.335499 4935 generic.go:356] "Generic (PLEG): container finished" podID="1ea9903b-325f-4281-92b9-e608ec727ea4" containerID="ea0771ddde4ef4fcd5870c257f7097b9ddd81367749429a657fc2d9925775a5a" exitCode=0 Nov 02 21:43:06.335759 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:06.335557 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f" Nov 02 21:43:06.335759 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:06.335578 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f" event={"ID":"1ea9903b-325f-4281-92b9-e608ec727ea4","Type":"ContainerDied","Data":"ea0771ddde4ef4fcd5870c257f7097b9ddd81367749429a657fc2d9925775a5a"} Nov 02 21:43:06.335759 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:06.335636 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f" event={"ID":"1ea9903b-325f-4281-92b9-e608ec727ea4","Type":"ContainerDied","Data":"da9e04e65b450a79a31018776add68e27e41093cc3393c632016421e97a2f6e2"} Nov 02 21:43:06.335759 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:06.335670 4935 scope.go:122] "RemoveContainer" containerID="ea0771ddde4ef4fcd5870c257f7097b9ddd81367749429a657fc2d9925775a5a" Nov 02 21:43:06.350799 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:06.349782 4935 scope.go:122] "RemoveContainer" containerID="ea0771ddde4ef4fcd5870c257f7097b9ddd81367749429a657fc2d9925775a5a" Nov 02 21:43:06.350799 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:43:06.350186 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea0771ddde4ef4fcd5870c257f7097b9ddd81367749429a657fc2d9925775a5a\": container with ID starting with ea0771ddde4ef4fcd5870c257f7097b9ddd81367749429a657fc2d9925775a5a not found: ID does not exist" containerID="ea0771ddde4ef4fcd5870c257f7097b9ddd81367749429a657fc2d9925775a5a" Nov 02 21:43:06.350799 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:06.350221 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea0771ddde4ef4fcd5870c257f7097b9ddd81367749429a657fc2d9925775a5a"} err="failed to get container status \"ea0771ddde4ef4fcd5870c257f7097b9ddd81367749429a657fc2d9925775a5a\": rpc error: code = NotFound desc = could not find container \"ea0771ddde4ef4fcd5870c257f7097b9ddd81367749429a657fc2d9925775a5a\": container with ID starting with ea0771ddde4ef4fcd5870c257f7097b9ddd81367749429a657fc2d9925775a5a not found: ID does not exist" Nov 02 21:43:06.492520 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:06.492488 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f"] Nov 02 21:43:06.537264 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:06.537231 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4453bd-5cd7cc6467-7vr9f"] Nov 02 21:43:07.508547 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:07.508507 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:07.778121 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:07.778088 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ea9903b-325f-4281-92b9-e608ec727ea4" path="/var/lib/kubelet/pods/1ea9903b-325f-4281-92b9-e608ec727ea4/volumes" Nov 02 21:43:08.169970 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:08.169863 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:08.403077 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:08.403042 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:09.782697 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:09.782646 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:10.327887 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:10.327848 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:10.554395 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:10.554364 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:10.776855 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:10.776824 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:11.802015 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:11.801966 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:12.218956 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.218869 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k"] Nov 02 21:43:12.219470 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.219443 4935 cpu_manager.go:414] "containerMap: removing container" podUID="45871915-74c7-4bf7-a338-31f16c746f76" containerName="maas-api" Nov 02 21:43:12.219470 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.219461 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45871915-74c7-4bf7-a338-31f16c746f76" containerName="maas-api" Nov 02 21:43:12.219653 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.219491 4935 cpu_manager.go:414] "containerMap: removing container" podUID="1ea9903b-325f-4281-92b9-e608ec727ea4" containerName="maas-api" Nov 02 21:43:12.219653 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.219500 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea9903b-325f-4281-92b9-e608ec727ea4" containerName="maas-api" Nov 02 21:43:12.219653 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.219601 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea9903b-325f-4281-92b9-e608ec727ea4" containerName="maas-api" Nov 02 21:43:12.219653 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.219620 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="45871915-74c7-4bf7-a338-31f16c746f76" containerName="maas-api" Nov 02 21:43:12.224353 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.224333 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.316222 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.314967 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istio-token\") pod \"e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.316731 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.316621 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-credential-socket\") pod \"e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.317001 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.316980 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istio-envoy\") pod \"e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.317178 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.317158 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-workload-socket\") pod \"e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.317304 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.317289 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-workload-certs\") pod \"e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.317498 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.317469 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istio-podinfo\") pod \"e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.317772 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.317744 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istiod-ca-cert\") pod \"e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.317920 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.317897 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istio-data\") pod \"e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.318076 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.318060 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vcx8\" (UniqueName: \"kubernetes.io/projected/ff38001c-9b28-473a-bdc7-9ebc8514fe51-kube-api-access-9vcx8\") pod \"e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.332588 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.332555 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k"] Nov 02 21:43:12.376085 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.376050 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-be6df0-openshift-default-dockercfg-lh8mh\"" Nov 02 21:43:12.417937 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.417204 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:12.421366 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.420484 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istio-token\") pod \"e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.421366 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.420575 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-credential-socket\") pod \"e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.421366 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.420622 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istio-envoy\") pod \"e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.421366 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.420668 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-workload-socket\") pod \"e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.421366 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.420697 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-workload-certs\") pod \"e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.421366 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.420729 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istio-podinfo\") pod \"e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.421366 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.420804 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istiod-ca-cert\") pod \"e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.421366 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.420839 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istio-data\") pod \"e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.421366 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.420874 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vcx8\" (UniqueName: \"kubernetes.io/projected/ff38001c-9b28-473a-bdc7-9ebc8514fe51-kube-api-access-9vcx8\") pod \"e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.421366 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.421248 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-workload-socket\") pod \"e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.422070 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.421450 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istio-data\") pod \"e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.422070 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.421567 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-credential-socket\") pod \"e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.422070 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.422032 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-workload-certs\") pod \"e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.422561 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.422540 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istiod-ca-cert\") pod \"e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.424473 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.424446 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istio-envoy\") pod \"e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.437717 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.437656 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istio-podinfo\") pod \"e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.474461 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.474389 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istio-token\") pod \"e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.474600 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.474464 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vcx8\" (UniqueName: \"kubernetes.io/projected/ff38001c-9b28-473a-bdc7-9ebc8514fe51-kube-api-access-9vcx8\") pod \"e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.540733 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.540696 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:12.722070 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.722036 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:12.971802 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:12.971769 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k"] Nov 02 21:43:13.010951 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:43:13.010922 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff38001c_9b28_473a_bdc7_9ebc8514fe51.slice/crio-54ee0e4a6a0f02d89f5c3cc89f2e18951cb0182b464981e60bc5fe42d7b7165c WatchSource:0}: Error finding container 54ee0e4a6a0f02d89f5c3cc89f2e18951cb0182b464981e60bc5fe42d7b7165c: Status 404 returned error can't find the container with id 54ee0e4a6a0f02d89f5c3cc89f2e18951cb0182b464981e60bc5fe42d7b7165c Nov 02 21:43:13.014252 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:13.014208 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:43:13.014356 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:13.014304 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:43:13.014409 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:13.014350 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:43:13.391114 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:13.391072 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:13.401804 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:13.401763 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" event={"ID":"ff38001c-9b28-473a-bdc7-9ebc8514fe51","Type":"ContainerStarted","Data":"b6a1e2d518e2203d438d319e1c2cf68f7fb32bcf5500cb73b061397e98464da1"} Nov 02 21:43:13.401972 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:13.401831 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" event={"ID":"ff38001c-9b28-473a-bdc7-9ebc8514fe51","Type":"ContainerStarted","Data":"54ee0e4a6a0f02d89f5c3cc89f2e18951cb0182b464981e60bc5fe42d7b7165c"} Nov 02 21:43:13.512309 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:13.512253 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" podStartSLOduration=1.512238961 podStartE2EDuration="1.512238961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:43:13.509290948 +0000 UTC m=+2085.948865673" watchObservedRunningTime="2026-11-02 21:43:13.512238961 +0000 UTC m=+2085.951813745" Nov 02 21:43:13.540816 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:13.540790 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:14.547875 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:14.547841 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:15.416385 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:15.416354 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:15.417254 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:15.417235 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:43:16.375016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:16.374981 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:16.614469 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:16.614427 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:17.533778 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:17.533743 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:17.746799 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:17.746761 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:20.634585 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:20.634554 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:20.914088 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:20.914006 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:21.123866 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:21.123825 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:25.728768 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:25.728729 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:28.189170 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:28.189110 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 21:43:28.196926 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:28.196899 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 21:43:28.198464 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:28.198439 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 21:43:28.206683 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:28.206662 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 21:43:28.474288 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:43:28.474200 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:43:29.190692 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:29.190655 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:31.041281 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:31.041248 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:31.245929 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:31.245894 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:32.642347 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:32.642307 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4"] Nov 02 21:43:32.740902 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:32.740866 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch"] Nov 02 21:43:33.065964 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:33.065922 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:33.270692 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:33.270651 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:36.338888 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:36.338856 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:37.643728 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:37.643678 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4" podUID="a6f83b79-d7fe-49b7-9a9e-036522f072ab" containerName="payload-processing" containerID="cri-o://5b20c9f8c7250307919f988bde4df564d4c98d1a12eadecf326f9e50a8749a39" gracePeriod=25 Nov 02 21:43:37.741388 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:37.741334 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch" podUID="1fb23512-885f-4a8d-9f2c-4e6819e3b8b4" containerName="payload-pre-processing" containerID="cri-o://3a610378faed6d413bc7641c4a6578cf4f479fa73a6b2e30a21f00f5672f2bfb" gracePeriod=25 Nov 02 21:43:37.934183 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:37.933839 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4" Nov 02 21:43:38.026629 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.026604 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch" Nov 02 21:43:38.050379 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.050338 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6f83b79-d7fe-49b7-9a9e-036522f072ab-kube-api-access-p44tm\" (UniqueName: \"kubernetes.io/projected/a6f83b79-d7fe-49b7-9a9e-036522f072ab-kube-api-access-p44tm\") pod \"a6f83b79-d7fe-49b7-9a9e-036522f072ab\" (UID: \"a6f83b79-d7fe-49b7-9a9e-036522f072ab\") " Nov 02 21:43:38.050573 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.050543 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6f83b79-d7fe-49b7-9a9e-036522f072ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6f83b79-d7fe-49b7-9a9e-036522f072ab-plugins-config-volume\") pod \"a6f83b79-d7fe-49b7-9a9e-036522f072ab\" (UID: \"a6f83b79-d7fe-49b7-9a9e-036522f072ab\") " Nov 02 21:43:38.050981 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.050941 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f83b79-d7fe-49b7-9a9e-036522f072ab-plugins-config-volume" pod "a6f83b79-d7fe-49b7-9a9e-036522f072ab" (UID: "a6f83b79-d7fe-49b7-9a9e-036522f072ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:43:38.052754 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.052723 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f83b79-d7fe-49b7-9a9e-036522f072ab-kube-api-access-p44tm" pod "a6f83b79-d7fe-49b7-9a9e-036522f072ab" (UID: "a6f83b79-d7fe-49b7-9a9e-036522f072ab"). InnerVolumeSpecName "kube-api-access-p44tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:43:38.152899 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.152819 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb23512-885f-4a8d-9f2c-4e6819e3b8b4-kube-api-access-slf8r\" (UniqueName: \"kubernetes.io/projected/1fb23512-885f-4a8d-9f2c-4e6819e3b8b4-kube-api-access-slf8r\") pod \"1fb23512-885f-4a8d-9f2c-4e6819e3b8b4\" (UID: \"1fb23512-885f-4a8d-9f2c-4e6819e3b8b4\") " Nov 02 21:43:38.153048 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.152953 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fb23512-885f-4a8d-9f2c-4e6819e3b8b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fb23512-885f-4a8d-9f2c-4e6819e3b8b4-plugins-config-volume\") pod \"1fb23512-885f-4a8d-9f2c-4e6819e3b8b4\" (UID: \"1fb23512-885f-4a8d-9f2c-4e6819e3b8b4\") " Nov 02 21:43:38.153575 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.153523 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p44tm\" (UniqueName: \"kubernetes.io/projected/a6f83b79-d7fe-49b7-9a9e-036522f072ab-kube-api-access-p44tm\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:38.153575 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.153551 4935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6f83b79-d7fe-49b7-9a9e-036522f072ab-plugins-config-volume\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:38.154031 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.153788 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fb23512-885f-4a8d-9f2c-4e6819e3b8b4-plugins-config-volume" pod "1fb23512-885f-4a8d-9f2c-4e6819e3b8b4" (UID: "1fb23512-885f-4a8d-9f2c-4e6819e3b8b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:43:38.155495 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.155471 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb23512-885f-4a8d-9f2c-4e6819e3b8b4-kube-api-access-slf8r" pod "1fb23512-885f-4a8d-9f2c-4e6819e3b8b4" (UID: "1fb23512-885f-4a8d-9f2c-4e6819e3b8b4"). InnerVolumeSpecName "kube-api-access-slf8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:43:38.254526 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.254484 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slf8r\" (UniqueName: \"kubernetes.io/projected/1fb23512-885f-4a8d-9f2c-4e6819e3b8b4-kube-api-access-slf8r\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:38.254526 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.254525 4935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fb23512-885f-4a8d-9f2c-4e6819e3b8b4-plugins-config-volume\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:38.571745 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.571705 4935 generic.go:356] "Generic (PLEG): container finished" podID="a6f83b79-d7fe-49b7-9a9e-036522f072ab" containerID="5b20c9f8c7250307919f988bde4df564d4c98d1a12eadecf326f9e50a8749a39" exitCode=0 Nov 02 21:43:38.571947 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.571775 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4" Nov 02 21:43:38.571947 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.571787 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4" event={"ID":"a6f83b79-d7fe-49b7-9a9e-036522f072ab","Type":"ContainerDied","Data":"5b20c9f8c7250307919f988bde4df564d4c98d1a12eadecf326f9e50a8749a39"} Nov 02 21:43:38.571947 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.571874 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4" event={"ID":"a6f83b79-d7fe-49b7-9a9e-036522f072ab","Type":"ContainerDied","Data":"63a7bac6a0005fe158d0fa381a23c0552be8217677b340dcc8cef50ff66fad45"} Nov 02 21:43:38.571947 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.571902 4935 scope.go:122] "RemoveContainer" containerID="5b20c9f8c7250307919f988bde4df564d4c98d1a12eadecf326f9e50a8749a39" Nov 02 21:43:38.574351 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.573200 4935 generic.go:356] "Generic (PLEG): container finished" podID="1fb23512-885f-4a8d-9f2c-4e6819e3b8b4" containerID="3a610378faed6d413bc7641c4a6578cf4f479fa73a6b2e30a21f00f5672f2bfb" exitCode=0 Nov 02 21:43:38.574351 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.573234 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch" event={"ID":"1fb23512-885f-4a8d-9f2c-4e6819e3b8b4","Type":"ContainerDied","Data":"3a610378faed6d413bc7641c4a6578cf4f479fa73a6b2e30a21f00f5672f2bfb"} Nov 02 21:43:38.574351 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.573268 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch" event={"ID":"1fb23512-885f-4a8d-9f2c-4e6819e3b8b4","Type":"ContainerDied","Data":"86ae4eec8090a6d55fc848c8dbfee4afddcc731f819ef2da1ccae99b47c47d6a"} Nov 02 21:43:38.574351 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.573276 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch" Nov 02 21:43:38.584604 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.584579 4935 scope.go:122] "RemoveContainer" containerID="5b20c9f8c7250307919f988bde4df564d4c98d1a12eadecf326f9e50a8749a39" Nov 02 21:43:38.585179 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:43:38.585114 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b20c9f8c7250307919f988bde4df564d4c98d1a12eadecf326f9e50a8749a39\": container with ID starting with 5b20c9f8c7250307919f988bde4df564d4c98d1a12eadecf326f9e50a8749a39 not found: ID does not exist" containerID="5b20c9f8c7250307919f988bde4df564d4c98d1a12eadecf326f9e50a8749a39" Nov 02 21:43:38.585296 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.585189 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b20c9f8c7250307919f988bde4df564d4c98d1a12eadecf326f9e50a8749a39"} err="failed to get container status \"5b20c9f8c7250307919f988bde4df564d4c98d1a12eadecf326f9e50a8749a39\": rpc error: code = NotFound desc = could not find container \"5b20c9f8c7250307919f988bde4df564d4c98d1a12eadecf326f9e50a8749a39\": container with ID starting with 5b20c9f8c7250307919f988bde4df564d4c98d1a12eadecf326f9e50a8749a39 not found: ID does not exist" Nov 02 21:43:38.585296 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.585214 4935 scope.go:122] "RemoveContainer" containerID="3a610378faed6d413bc7641c4a6578cf4f479fa73a6b2e30a21f00f5672f2bfb" Nov 02 21:43:38.596190 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.596165 4935 scope.go:122] "RemoveContainer" containerID="3a610378faed6d413bc7641c4a6578cf4f479fa73a6b2e30a21f00f5672f2bfb" Nov 02 21:43:38.596525 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:43:38.596504 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a610378faed6d413bc7641c4a6578cf4f479fa73a6b2e30a21f00f5672f2bfb\": container with ID starting with 3a610378faed6d413bc7641c4a6578cf4f479fa73a6b2e30a21f00f5672f2bfb not found: ID does not exist" containerID="3a610378faed6d413bc7641c4a6578cf4f479fa73a6b2e30a21f00f5672f2bfb" Nov 02 21:43:38.596608 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.596533 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a610378faed6d413bc7641c4a6578cf4f479fa73a6b2e30a21f00f5672f2bfb"} err="failed to get container status \"3a610378faed6d413bc7641c4a6578cf4f479fa73a6b2e30a21f00f5672f2bfb\": rpc error: code = NotFound desc = could not find container \"3a610378faed6d413bc7641c4a6578cf4f479fa73a6b2e30a21f00f5672f2bfb\": container with ID starting with 3a610378faed6d413bc7641c4a6578cf4f479fa73a6b2e30a21f00f5672f2bfb not found: ID does not exist" Nov 02 21:43:38.727009 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.726975 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4"] Nov 02 21:43:38.773309 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.773270 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3e3997-679f874fdc-ck8f4"] Nov 02 21:43:38.875068 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.874971 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch"] Nov 02 21:43:38.877830 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:38.877805 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3997-75757b575-mxpch"] Nov 02 21:43:39.641806 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.641764 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf"] Nov 02 21:43:39.642410 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.642389 4935 cpu_manager.go:414] "containerMap: removing container" podUID="a6f83b79-d7fe-49b7-9a9e-036522f072ab" containerName="payload-processing" Nov 02 21:43:39.642537 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.642410 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6f83b79-d7fe-49b7-9a9e-036522f072ab" containerName="payload-processing" Nov 02 21:43:39.642537 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.642430 4935 cpu_manager.go:414] "containerMap: removing container" podUID="1fb23512-885f-4a8d-9f2c-4e6819e3b8b4" containerName="payload-pre-processing" Nov 02 21:43:39.642537 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.642438 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fb23512-885f-4a8d-9f2c-4e6819e3b8b4" containerName="payload-pre-processing" Nov 02 21:43:39.642699 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.642591 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6f83b79-d7fe-49b7-9a9e-036522f072ab" containerName="payload-processing" Nov 02 21:43:39.642699 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.642608 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fb23512-885f-4a8d-9f2c-4e6819e3b8b4" containerName="payload-pre-processing" Nov 02 21:43:39.647423 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.647400 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" Nov 02 21:43:39.700710 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.700669 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf"] Nov 02 21:43:39.740672 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.740636 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-e12f99\"/\"kube-root-ca.crt\"" Nov 02 21:43:39.741066 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.740704 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-e12f99\"/\"openshift-service-ca.crt\"" Nov 02 21:43:39.741066 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.740797 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-e12f99\"/\"ipp-route-e12f99-kserve-self-signed-certs\"" Nov 02 21:43:39.742545 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.742528 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-e12f99\"/\"default-dockercfg-xg84n\"" Nov 02 21:43:39.774828 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.774797 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6676be94-69f0-477e-9423-d45d58eb0d35-tmp-dir\") pod \"ipp-route-e12f99-kserve-75844fbb5c-7q5jf\" (UID: \"6676be94-69f0-477e-9423-d45d58eb0d35\") " pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" Nov 02 21:43:39.774989 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.774911 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwn8j\" (UniqueName: \"kubernetes.io/projected/6676be94-69f0-477e-9423-d45d58eb0d35-kube-api-access-hwn8j\") pod \"ipp-route-e12f99-kserve-75844fbb5c-7q5jf\" (UID: \"6676be94-69f0-477e-9423-d45d58eb0d35\") " pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" Nov 02 21:43:39.775067 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.775029 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6676be94-69f0-477e-9423-d45d58eb0d35-dshm\") pod \"ipp-route-e12f99-kserve-75844fbb5c-7q5jf\" (UID: \"6676be94-69f0-477e-9423-d45d58eb0d35\") " pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" Nov 02 21:43:39.775109 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.775066 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6676be94-69f0-477e-9423-d45d58eb0d35-tls-certs\") pod \"ipp-route-e12f99-kserve-75844fbb5c-7q5jf\" (UID: \"6676be94-69f0-477e-9423-d45d58eb0d35\") " pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" Nov 02 21:43:39.775199 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.775184 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6676be94-69f0-477e-9423-d45d58eb0d35-home\") pod \"ipp-route-e12f99-kserve-75844fbb5c-7q5jf\" (UID: \"6676be94-69f0-477e-9423-d45d58eb0d35\") " pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" Nov 02 21:43:39.775266 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.775252 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6676be94-69f0-477e-9423-d45d58eb0d35-model-cache\") pod \"ipp-route-e12f99-kserve-75844fbb5c-7q5jf\" (UID: \"6676be94-69f0-477e-9423-d45d58eb0d35\") " pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" Nov 02 21:43:39.777752 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.777719 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fb23512-885f-4a8d-9f2c-4e6819e3b8b4" path="/var/lib/kubelet/pods/1fb23512-885f-4a8d-9f2c-4e6819e3b8b4/volumes" Nov 02 21:43:39.778422 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.778402 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6f83b79-d7fe-49b7-9a9e-036522f072ab" path="/var/lib/kubelet/pods/a6f83b79-d7fe-49b7-9a9e-036522f072ab/volumes" Nov 02 21:43:39.817447 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.817403 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:39.876641 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.876596 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwn8j\" (UniqueName: \"kubernetes.io/projected/6676be94-69f0-477e-9423-d45d58eb0d35-kube-api-access-hwn8j\") pod \"ipp-route-e12f99-kserve-75844fbb5c-7q5jf\" (UID: \"6676be94-69f0-477e-9423-d45d58eb0d35\") " pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" Nov 02 21:43:39.876862 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.876702 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6676be94-69f0-477e-9423-d45d58eb0d35-dshm\") pod \"ipp-route-e12f99-kserve-75844fbb5c-7q5jf\" (UID: \"6676be94-69f0-477e-9423-d45d58eb0d35\") " pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" Nov 02 21:43:39.876862 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.876736 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6676be94-69f0-477e-9423-d45d58eb0d35-tls-certs\") pod \"ipp-route-e12f99-kserve-75844fbb5c-7q5jf\" (UID: \"6676be94-69f0-477e-9423-d45d58eb0d35\") " pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" Nov 02 21:43:39.876862 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.876795 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6676be94-69f0-477e-9423-d45d58eb0d35-home\") pod \"ipp-route-e12f99-kserve-75844fbb5c-7q5jf\" (UID: \"6676be94-69f0-477e-9423-d45d58eb0d35\") " pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" Nov 02 21:43:39.877028 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.876890 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6676be94-69f0-477e-9423-d45d58eb0d35-model-cache\") pod \"ipp-route-e12f99-kserve-75844fbb5c-7q5jf\" (UID: \"6676be94-69f0-477e-9423-d45d58eb0d35\") " pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" Nov 02 21:43:39.877028 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.876980 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6676be94-69f0-477e-9423-d45d58eb0d35-tmp-dir\") pod \"ipp-route-e12f99-kserve-75844fbb5c-7q5jf\" (UID: \"6676be94-69f0-477e-9423-d45d58eb0d35\") " pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" Nov 02 21:43:39.877492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.877468 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6676be94-69f0-477e-9423-d45d58eb0d35-tmp-dir\") pod \"ipp-route-e12f99-kserve-75844fbb5c-7q5jf\" (UID: \"6676be94-69f0-477e-9423-d45d58eb0d35\") " pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" Nov 02 21:43:39.877612 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.877529 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6676be94-69f0-477e-9423-d45d58eb0d35-model-cache\") pod \"ipp-route-e12f99-kserve-75844fbb5c-7q5jf\" (UID: \"6676be94-69f0-477e-9423-d45d58eb0d35\") " pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" Nov 02 21:43:39.877612 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.877602 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6676be94-69f0-477e-9423-d45d58eb0d35-home\") pod \"ipp-route-e12f99-kserve-75844fbb5c-7q5jf\" (UID: \"6676be94-69f0-477e-9423-d45d58eb0d35\") " pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" Nov 02 21:43:39.879792 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.879764 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6676be94-69f0-477e-9423-d45d58eb0d35-dshm\") pod \"ipp-route-e12f99-kserve-75844fbb5c-7q5jf\" (UID: \"6676be94-69f0-477e-9423-d45d58eb0d35\") " pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" Nov 02 21:43:39.880332 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.880310 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6676be94-69f0-477e-9423-d45d58eb0d35-tls-certs\") pod \"ipp-route-e12f99-kserve-75844fbb5c-7q5jf\" (UID: \"6676be94-69f0-477e-9423-d45d58eb0d35\") " pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" Nov 02 21:43:39.939199 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.930777 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwn8j\" (UniqueName: \"kubernetes.io/projected/6676be94-69f0-477e-9423-d45d58eb0d35-kube-api-access-hwn8j\") pod \"ipp-route-e12f99-kserve-75844fbb5c-7q5jf\" (UID: \"6676be94-69f0-477e-9423-d45d58eb0d35\") " pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" Nov 02 21:43:39.961193 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:39.961161 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" Nov 02 21:43:40.344413 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:40.344380 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf"] Nov 02 21:43:40.344659 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:43:40.344635 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6676be94_69f0_477e_9423_d45d58eb0d35.slice/crio-2e26feb29f413c53d91153ae366142774d73a2feb2f70bb6840eba36346e07b7 WatchSource:0}: Error finding container 2e26feb29f413c53d91153ae366142774d73a2feb2f70bb6840eba36346e07b7: Status 404 returned error can't find the container with id 2e26feb29f413c53d91153ae366142774d73a2feb2f70bb6840eba36346e07b7 Nov 02 21:43:40.591188 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:40.591124 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" event={"ID":"6676be94-69f0-477e-9423-d45d58eb0d35","Type":"ContainerStarted","Data":"da883ae9608c3eaade8491008a575e4dbc53a99c2b96cfb0033a8ee36c6c1cf7"} Nov 02 21:43:40.591406 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:40.591202 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" event={"ID":"6676be94-69f0-477e-9423-d45d58eb0d35","Type":"ContainerStarted","Data":"2e26feb29f413c53d91153ae366142774d73a2feb2f70bb6840eba36346e07b7"} Nov 02 21:43:40.701038 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:40.700976 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" podStartSLOduration=1.700953244 podStartE2EDuration="1.700953244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:43:40.699509771 +0000 UTC m=+2113.139084494" watchObservedRunningTime="2026-11-02 21:43:40.700953244 +0000 UTC m=+2113.140527965" Nov 02 21:43:40.961826 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:40.961749 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" Nov 02 21:43:41.302772 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:41.302730 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:46.532287 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:46.532249 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:49.961528 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:49.961490 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" Nov 02 21:43:49.974173 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:49.974123 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" Nov 02 21:43:50.227095 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:50.227007 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:50.667953 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:50.667916 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" Nov 02 21:43:50.722187 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:50.722149 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:53.735089 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:43:53.735051 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:43:58.469593 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:43:58.469551 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:44:00.419403 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:00.419369 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:05.442106 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:05.442069 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:06.033097 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:06.033063 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:08.826356 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:08.826320 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:10.634010 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:10.633970 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:14.723684 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:14.723631 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:17.782486 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:17.782444 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:21.653949 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:21.653913 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:22.141177 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:22.140542 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:22.722548 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:22.722511 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:22.927165 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:22.927116 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:25.193743 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:25.193700 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:28.474787 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:44:28.474749 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:44:28.804512 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:28.804471 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:29.315199 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:29.315100 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:29.520504 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:29.520461 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:31.310443 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:31.310405 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9"] Nov 02 21:44:31.310981 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:31.310642 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9" podUID="e29452f7-d10d-4157-9789-f98c9111d8e8" containerName="maas-api" containerID="cri-o://7466d7cbb210860dceb1a0a567310986878792a98cb5d4dfa71f869f1df17396" gracePeriod=30 Nov 02 21:44:31.484529 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:31.484490 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:31.672567 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:31.672532 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9" Nov 02 21:44:31.742641 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:31.742592 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e29452f7-d10d-4157-9789-f98c9111d8e8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e29452f7-d10d-4157-9789-f98c9111d8e8-metrics-certs\") pod \"e29452f7-d10d-4157-9789-f98c9111d8e8\" (UID: \"e29452f7-d10d-4157-9789-f98c9111d8e8\") " Nov 02 21:44:31.742976 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:31.742952 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e29452f7-d10d-4157-9789-f98c9111d8e8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e29452f7-d10d-4157-9789-f98c9111d8e8-maas-api-tls\") pod \"e29452f7-d10d-4157-9789-f98c9111d8e8\" (UID: \"e29452f7-d10d-4157-9789-f98c9111d8e8\") " Nov 02 21:44:31.743372 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:31.743348 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e29452f7-d10d-4157-9789-f98c9111d8e8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e29452f7-d10d-4157-9789-f98c9111d8e8-openshift-service-ca\") pod \"e29452f7-d10d-4157-9789-f98c9111d8e8\" (UID: \"e29452f7-d10d-4157-9789-f98c9111d8e8\") " Nov 02 21:44:31.743566 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:31.743550 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e29452f7-d10d-4157-9789-f98c9111d8e8-kube-api-access-5xl7k\" (UniqueName: \"kubernetes.io/projected/e29452f7-d10d-4157-9789-f98c9111d8e8-kube-api-access-5xl7k\") pod \"e29452f7-d10d-4157-9789-f98c9111d8e8\" (UID: \"e29452f7-d10d-4157-9789-f98c9111d8e8\") " Nov 02 21:44:31.754608 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:31.754567 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e29452f7-d10d-4157-9789-f98c9111d8e8-openshift-service-ca" pod "e29452f7-d10d-4157-9789-f98c9111d8e8" (UID: "e29452f7-d10d-4157-9789-f98c9111d8e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:44:31.771625 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:31.761295 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e29452f7-d10d-4157-9789-f98c9111d8e8-metrics-certs" pod "e29452f7-d10d-4157-9789-f98c9111d8e8" (UID: "e29452f7-d10d-4157-9789-f98c9111d8e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:44:31.771625 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:31.761434 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e29452f7-d10d-4157-9789-f98c9111d8e8-kube-api-access-5xl7k" pod "e29452f7-d10d-4157-9789-f98c9111d8e8" (UID: "e29452f7-d10d-4157-9789-f98c9111d8e8"). InnerVolumeSpecName "kube-api-access-5xl7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:44:31.771625 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:31.761685 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e29452f7-d10d-4157-9789-f98c9111d8e8-maas-api-tls" pod "e29452f7-d10d-4157-9789-f98c9111d8e8" (UID: "e29452f7-d10d-4157-9789-f98c9111d8e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:44:31.845604 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:31.845571 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e29452f7-d10d-4157-9789-f98c9111d8e8-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:44:31.845604 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:31.845601 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e29452f7-d10d-4157-9789-f98c9111d8e8-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:44:31.845810 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:31.845614 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e29452f7-d10d-4157-9789-f98c9111d8e8-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:44:31.845810 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:31.845625 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xl7k\" (UniqueName: \"kubernetes.io/projected/e29452f7-d10d-4157-9789-f98c9111d8e8-kube-api-access-5xl7k\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:44:31.932725 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:31.932638 4935 generic.go:356] "Generic (PLEG): container finished" podID="e29452f7-d10d-4157-9789-f98c9111d8e8" containerID="7466d7cbb210860dceb1a0a567310986878792a98cb5d4dfa71f869f1df17396" exitCode=0 Nov 02 21:44:31.932725 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:31.932684 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9" Nov 02 21:44:31.932903 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:31.932732 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9" event={"ID":"e29452f7-d10d-4157-9789-f98c9111d8e8","Type":"ContainerDied","Data":"7466d7cbb210860dceb1a0a567310986878792a98cb5d4dfa71f869f1df17396"} Nov 02 21:44:31.932903 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:31.932801 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9" event={"ID":"e29452f7-d10d-4157-9789-f98c9111d8e8","Type":"ContainerDied","Data":"529bc89288835fb8c122c633e100c2bc6fd1f2211801c01fa1480b4b7b6b24db"} Nov 02 21:44:31.932903 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:31.932831 4935 scope.go:122] "RemoveContainer" containerID="7466d7cbb210860dceb1a0a567310986878792a98cb5d4dfa71f869f1df17396" Nov 02 21:44:31.953335 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:31.953302 4935 scope.go:122] "RemoveContainer" containerID="7466d7cbb210860dceb1a0a567310986878792a98cb5d4dfa71f869f1df17396" Nov 02 21:44:31.955074 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:44:31.955041 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7466d7cbb210860dceb1a0a567310986878792a98cb5d4dfa71f869f1df17396\": container with ID starting with 7466d7cbb210860dceb1a0a567310986878792a98cb5d4dfa71f869f1df17396 not found: ID does not exist" containerID="7466d7cbb210860dceb1a0a567310986878792a98cb5d4dfa71f869f1df17396" Nov 02 21:44:31.955201 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:31.955084 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7466d7cbb210860dceb1a0a567310986878792a98cb5d4dfa71f869f1df17396"} err="failed to get container status \"7466d7cbb210860dceb1a0a567310986878792a98cb5d4dfa71f869f1df17396\": rpc error: code = NotFound desc = could not find container \"7466d7cbb210860dceb1a0a567310986878792a98cb5d4dfa71f869f1df17396\": container with ID starting with 7466d7cbb210860dceb1a0a567310986878792a98cb5d4dfa71f869f1df17396 not found: ID does not exist" Nov 02 21:44:32.006971 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:32.006939 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:32.086886 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:32.086853 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9"] Nov 02 21:44:32.092471 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:32.092441 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-15253e-f8c5dddd5-kwld9"] Nov 02 21:44:32.217984 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:32.217905 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:33.021239 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:33.021201 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd"] Nov 02 21:44:33.021886 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:33.021865 4935 cpu_manager.go:414] "containerMap: removing container" podUID="e29452f7-d10d-4157-9789-f98c9111d8e8" containerName="maas-api" Nov 02 21:44:33.021938 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:33.021889 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e29452f7-d10d-4157-9789-f98c9111d8e8" containerName="maas-api" Nov 02 21:44:33.021998 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:33.021988 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="e29452f7-d10d-4157-9789-f98c9111d8e8" containerName="maas-api" Nov 02 21:44:33.026856 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:33.026833 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd" Nov 02 21:44:33.074859 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:33.074828 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a273e83e-21f3-4f31-babb-94579b007a61-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd\" (UID: \"a273e83e-21f3-4f31-babb-94579b007a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd" Nov 02 21:44:33.075052 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:33.074955 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxlfd\" (UniqueName: \"kubernetes.io/projected/a273e83e-21f3-4f31-babb-94579b007a61-kube-api-access-qxlfd\") pod \"maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd\" (UID: \"a273e83e-21f3-4f31-babb-94579b007a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd" Nov 02 21:44:33.075121 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:33.075100 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a273e83e-21f3-4f31-babb-94579b007a61-metrics-certs\") pod \"maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd\" (UID: \"a273e83e-21f3-4f31-babb-94579b007a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd" Nov 02 21:44:33.075240 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:33.075219 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a273e83e-21f3-4f31-babb-94579b007a61-maas-api-tls\") pod \"maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd\" (UID: \"a273e83e-21f3-4f31-babb-94579b007a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd" Nov 02 21:44:33.075644 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:33.075618 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-be6df0-serving-cert\"" Nov 02 21:44:33.079463 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:33.079438 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd"] Nov 02 21:44:33.176448 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:33.176408 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxlfd\" (UniqueName: \"kubernetes.io/projected/a273e83e-21f3-4f31-babb-94579b007a61-kube-api-access-qxlfd\") pod \"maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd\" (UID: \"a273e83e-21f3-4f31-babb-94579b007a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd" Nov 02 21:44:33.176638 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:33.176531 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a273e83e-21f3-4f31-babb-94579b007a61-metrics-certs\") pod \"maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd\" (UID: \"a273e83e-21f3-4f31-babb-94579b007a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd" Nov 02 21:44:33.176638 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:33.176589 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a273e83e-21f3-4f31-babb-94579b007a61-maas-api-tls\") pod \"maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd\" (UID: \"a273e83e-21f3-4f31-babb-94579b007a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd" Nov 02 21:44:33.176638 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:33.176609 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a273e83e-21f3-4f31-babb-94579b007a61-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd\" (UID: \"a273e83e-21f3-4f31-babb-94579b007a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd" Nov 02 21:44:33.176905 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:44:33.176883 4935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-serving-cert: secret "maas-api-e2e-praxis-a-be6df0-serving-cert" not found Nov 02 21:44:33.177012 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:44:33.176960 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a273e83e-21f3-4f31-babb-94579b007a61-maas-api-tls podName:a273e83e-21f3-4f31-babb-94579b007a61 nodeName:}" failed. No retries permitted until 2026-11-02 21:44:33.676937669 +0000 UTC m=+2166.116512372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a273e83e-21f3-4f31-babb-94579b007a61-maas-api-tls") pod "maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd" (UID: "a273e83e-21f3-4f31-babb-94579b007a61") : secret "maas-api-e2e-praxis-a-be6df0-serving-cert" not found Nov 02 21:44:33.177415 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:33.177390 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a273e83e-21f3-4f31-babb-94579b007a61-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd\" (UID: \"a273e83e-21f3-4f31-babb-94579b007a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd" Nov 02 21:44:33.179421 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:33.179396 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a273e83e-21f3-4f31-babb-94579b007a61-metrics-certs\") pod \"maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd\" (UID: \"a273e83e-21f3-4f31-babb-94579b007a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd" Nov 02 21:44:33.230747 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:33.230721 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxlfd\" (UniqueName: \"kubernetes.io/projected/a273e83e-21f3-4f31-babb-94579b007a61-kube-api-access-qxlfd\") pod \"maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd\" (UID: \"a273e83e-21f3-4f31-babb-94579b007a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd" Nov 02 21:44:33.287947 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:33.283624 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:33.484188 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:33.484122 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:33.682923 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:33.682836 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a273e83e-21f3-4f31-babb-94579b007a61-maas-api-tls\") pod \"maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd\" (UID: \"a273e83e-21f3-4f31-babb-94579b007a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd" Nov 02 21:44:33.685376 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:33.685348 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a273e83e-21f3-4f31-babb-94579b007a61-maas-api-tls\") pod \"maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd\" (UID: \"a273e83e-21f3-4f31-babb-94579b007a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd" Nov 02 21:44:33.776997 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:33.776960 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e29452f7-d10d-4157-9789-f98c9111d8e8" path="/var/lib/kubelet/pods/e29452f7-d10d-4157-9789-f98c9111d8e8/volumes" Nov 02 21:44:33.783478 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:33.783452 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:33.940306 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:33.940212 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd" Nov 02 21:44:34.310559 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:34.310530 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd"] Nov 02 21:44:34.311848 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:44:34.311821 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda273e83e_21f3_4f31_babb_94579b007a61.slice/crio-90556fc66239a46af90fe0e1bfd83d18c4a4815050dc271aafbb1fc5125f0f42 WatchSource:0}: Error finding container 90556fc66239a46af90fe0e1bfd83d18c4a4815050dc271aafbb1fc5125f0f42: Status 404 returned error can't find the container with id 90556fc66239a46af90fe0e1bfd83d18c4a4815050dc271aafbb1fc5125f0f42 Nov 02 21:44:34.313510 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:34.313487 4935 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:44:34.955085 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:34.955049 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd" event={"ID":"a273e83e-21f3-4f31-babb-94579b007a61","Type":"ContainerStarted","Data":"90556fc66239a46af90fe0e1bfd83d18c4a4815050dc271aafbb1fc5125f0f42"} Nov 02 21:44:35.965633 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:35.965595 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd" event={"ID":"a273e83e-21f3-4f31-babb-94579b007a61","Type":"ContainerStarted","Data":"815a51ec5d841f5e51cb131effb3fb5673913a5729e333971a30da21494afdeb"} Nov 02 21:44:35.966118 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:35.965729 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd" Nov 02 21:44:36.088884 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:36.088817 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd" podStartSLOduration=3.051763507 podStartE2EDuration="4.088792404s" totalImagesPullingTime="1.037028897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:44:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:44:36.081987732 +0000 UTC m=+2168.521562455" watchObservedRunningTime="2026-11-02 21:44:36.088792404 +0000 UTC m=+2168.528367127" Nov 02 21:44:37.788582 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:37.788542 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:38.001919 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:38.001885 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:38.882996 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:38.882947 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:39.085068 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.085033 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:39.419170 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.419097 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4"] Nov 02 21:44:39.424080 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.424052 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.541087 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.540832 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-aa60bc-openshift-default-dockercfg-zrktw\"" Nov 02 21:44:39.554268 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.546607 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4"] Nov 02 21:44:39.559328 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.559281 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-workload-socket\") pod \"e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.559624 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.559588 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istio-data\") pod \"e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.559975 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.559951 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9qtv\" (UniqueName: \"kubernetes.io/projected/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-kube-api-access-c9qtv\") pod \"e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.560258 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.560237 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-workload-certs\") pod \"e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.560458 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.560442 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istio-envoy\") pod \"e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.560663 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.560645 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istiod-ca-cert\") pod \"e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.560846 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.560830 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istio-podinfo\") pod \"e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.566888 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.563244 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istio-token\") pod \"e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.566888 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.563374 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-credential-socket\") pod \"e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.594195 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.594144 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:39.668055 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.667998 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istiod-ca-cert\") pod \"e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.668366 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.668069 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istio-podinfo\") pod \"e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.668366 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.668123 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istio-token\") pod \"e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.668366 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.668209 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-credential-socket\") pod \"e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.668366 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.668267 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-workload-socket\") pod \"e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.668366 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.668309 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istio-data\") pod \"e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.668836 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.668371 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9qtv\" (UniqueName: \"kubernetes.io/projected/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-kube-api-access-c9qtv\") pod \"e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.668836 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.668424 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-workload-certs\") pod \"e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.668836 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.668485 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istio-envoy\") pod \"e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.669385 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.669328 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istio-data\") pod \"e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.669935 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.669907 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-credential-socket\") pod \"e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.670597 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.670571 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-workload-socket\") pod \"e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.670948 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.670922 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istiod-ca-cert\") pod \"e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.671233 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.671200 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-workload-certs\") pod \"e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.673956 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.673920 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istio-podinfo\") pod \"e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.675397 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.675375 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istio-envoy\") pod \"e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.724433 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.724396 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istio-token\") pod \"e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.726585 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.726553 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9qtv\" (UniqueName: \"kubernetes.io/projected/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-kube-api-access-c9qtv\") pod \"e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.740429 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.740398 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:39.810537 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:39.810495 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:40.015701 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:40.015669 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:40.227734 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:40.226491 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4"] Nov 02 21:44:40.242221 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:40.240966 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:40.253175 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:44:40.245656 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf9d00da_108f_4cd1_8ca8_5f89ba59c624.slice/crio-d62dc3a0d7b42b6a7729f15c116de4bc544b2ef3c7ff5e6a91462d5bf037a3bc WatchSource:0}: Error finding container d62dc3a0d7b42b6a7729f15c116de4bc544b2ef3c7ff5e6a91462d5bf037a3bc: Status 404 returned error can't find the container with id d62dc3a0d7b42b6a7729f15c116de4bc544b2ef3c7ff5e6a91462d5bf037a3bc Nov 02 21:44:40.274768 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:40.270041 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:44:40.274768 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:40.270162 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:44:40.274768 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:40.270203 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:44:41.006189 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:41.006114 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:41.008734 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:41.008702 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" event={"ID":"cf9d00da-108f-4cd1-8ca8-5f89ba59c624","Type":"ContainerStarted","Data":"e2f2517e6c5b39eed75d461e5d6f4bf0ee4fa50480678707a15766b101019baf"} Nov 02 21:44:41.008896 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:41.008763 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" event={"ID":"cf9d00da-108f-4cd1-8ca8-5f89ba59c624","Type":"ContainerStarted","Data":"d62dc3a0d7b42b6a7729f15c116de4bc544b2ef3c7ff5e6a91462d5bf037a3bc"} Nov 02 21:44:41.119308 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:41.119253 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" podStartSLOduration=2.119238596 podStartE2EDuration="2.119238596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:44:41.118087378 +0000 UTC m=+2173.557662100" watchObservedRunningTime="2026-11-02 21:44:41.119238596 +0000 UTC m=+2173.558813317" Nov 02 21:44:41.740932 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:41.740887 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:41.746860 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:41.746833 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:41.984119 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:41.984091 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd" Nov 02 21:44:42.015067 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:42.015040 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:42.016100 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:42.016080 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:44:42.916942 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:42.916890 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:43.425648 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:43.425615 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:43.952713 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:43.952680 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:44.172101 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:44.172059 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:46.458738 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:46.458704 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:46.816640 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:46.816602 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:47.194572 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:47.194490 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:47.392219 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:47.392186 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:47.639165 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:47.624957 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:52.101170 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:52.101089 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:52.564775 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:52.564737 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:57.576805 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:44:57.576720 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:44:58.479600 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:44:58.479566 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:45:00.185751 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.185710 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm"] Nov 02 21:45:00.189785 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.189763 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm" Nov 02 21:45:00.193522 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.193495 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc"] Nov 02 21:45:00.197346 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.197319 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc" Nov 02 21:45:00.202522 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.202500 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc"] Nov 02 21:45:00.206478 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.206453 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc" Nov 02 21:45:00.307442 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.307410 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f454e9fe-c1a8-4057-8afe-329450b2b037-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc\" (UID: \"f454e9fe-c1a8-4057-8afe-329450b2b037\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc" Nov 02 21:45:00.307658 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.307604 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee59c9a7-7469-4f26-af41-95f0bcef64f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm\" (UID: \"ee59c9a7-7469-4f26-af41-95f0bcef64f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm" Nov 02 21:45:00.307658 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.307643 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrppr\" (UniqueName: \"kubernetes.io/projected/ee59c9a7-7469-4f26-af41-95f0bcef64f6-kube-api-access-vrppr\") pod \"maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm\" (UID: \"ee59c9a7-7469-4f26-af41-95f0bcef64f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm" Nov 02 21:45:00.307786 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.307705 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxxd7\" (UniqueName: \"kubernetes.io/projected/be71f888-c746-460e-96a2-70a060697231-kube-api-access-hxxd7\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc\" (UID: \"be71f888-c746-460e-96a2-70a060697231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc" Nov 02 21:45:00.307886 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.307869 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc5pm\" (UniqueName: \"kubernetes.io/projected/f454e9fe-c1a8-4057-8afe-329450b2b037-kube-api-access-rc5pm\") pod \"maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc\" (UID: \"f454e9fe-c1a8-4057-8afe-329450b2b037\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc" Nov 02 21:45:00.307967 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.307954 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be71f888-c746-460e-96a2-70a060697231-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc\" (UID: \"be71f888-c746-460e-96a2-70a060697231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc" Nov 02 21:45:00.314560 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.314460 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc"] Nov 02 21:45:00.338210 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.338168 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc"] Nov 02 21:45:00.340384 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.340354 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\"" Nov 02 21:45:00.380814 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.380697 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm"] Nov 02 21:45:00.409108 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.409079 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee59c9a7-7469-4f26-af41-95f0bcef64f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm\" (UID: \"ee59c9a7-7469-4f26-af41-95f0bcef64f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm" Nov 02 21:45:00.409294 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.409115 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrppr\" (UniqueName: \"kubernetes.io/projected/ee59c9a7-7469-4f26-af41-95f0bcef64f6-kube-api-access-vrppr\") pod \"maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm\" (UID: \"ee59c9a7-7469-4f26-af41-95f0bcef64f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm" Nov 02 21:45:00.409294 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.409168 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxxd7\" (UniqueName: \"kubernetes.io/projected/be71f888-c746-460e-96a2-70a060697231-kube-api-access-hxxd7\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc\" (UID: \"be71f888-c746-460e-96a2-70a060697231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc" Nov 02 21:45:00.409294 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.409198 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc5pm\" (UniqueName: \"kubernetes.io/projected/f454e9fe-c1a8-4057-8afe-329450b2b037-kube-api-access-rc5pm\") pod \"maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc\" (UID: \"f454e9fe-c1a8-4057-8afe-329450b2b037\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc" Nov 02 21:45:00.409294 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.409242 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be71f888-c746-460e-96a2-70a060697231-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc\" (UID: \"be71f888-c746-460e-96a2-70a060697231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc" Nov 02 21:45:00.409294 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.409279 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f454e9fe-c1a8-4057-8afe-329450b2b037-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc\" (UID: \"f454e9fe-c1a8-4057-8afe-329450b2b037\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc" Nov 02 21:45:00.410003 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.409981 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f454e9fe-c1a8-4057-8afe-329450b2b037-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc\" (UID: \"f454e9fe-c1a8-4057-8afe-329450b2b037\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc" Nov 02 21:45:00.410059 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.410032 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be71f888-c746-460e-96a2-70a060697231-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc\" (UID: \"be71f888-c746-460e-96a2-70a060697231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc" Nov 02 21:45:00.410213 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.410196 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee59c9a7-7469-4f26-af41-95f0bcef64f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm\" (UID: \"ee59c9a7-7469-4f26-af41-95f0bcef64f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm" Nov 02 21:45:00.463152 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.463102 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxxd7\" (UniqueName: \"kubernetes.io/projected/be71f888-c746-460e-96a2-70a060697231-kube-api-access-hxxd7\") pod \"maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc\" (UID: \"be71f888-c746-460e-96a2-70a060697231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc" Nov 02 21:45:00.463503 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.463479 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrppr\" (UniqueName: \"kubernetes.io/projected/ee59c9a7-7469-4f26-af41-95f0bcef64f6-kube-api-access-vrppr\") pod \"maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm\" (UID: \"ee59c9a7-7469-4f26-af41-95f0bcef64f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm" Nov 02 21:45:00.463602 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.463481 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc5pm\" (UniqueName: \"kubernetes.io/projected/f454e9fe-c1a8-4057-8afe-329450b2b037-kube-api-access-rc5pm\") pod \"maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc\" (UID: \"f454e9fe-c1a8-4057-8afe-329450b2b037\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc" Nov 02 21:45:00.506431 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.506399 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm" Nov 02 21:45:00.515709 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.515680 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc" Nov 02 21:45:00.522718 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.522692 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc" Nov 02 21:45:00.915311 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.915226 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc"] Nov 02 21:45:00.956788 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:45:00.956747 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe71f888_c746_460e_96a2_70a060697231.slice/crio-405bf4da76a7fb60490ec0a1b67a510382d1197d7aa02ee27646b9e071309602 WatchSource:0}: Error finding container 405bf4da76a7fb60490ec0a1b67a510382d1197d7aa02ee27646b9e071309602: Status 404 returned error can't find the container with id 405bf4da76a7fb60490ec0a1b67a510382d1197d7aa02ee27646b9e071309602 Nov 02 21:45:00.964185 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:45:00.964078 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf454e9fe_c1a8_4057_8afe_329450b2b037.slice/crio-36cbb1b18dbd4874ba6c867b2f7ff717f6c3f665bd1ae820fbc1e258c1761425 WatchSource:0}: Error finding container 36cbb1b18dbd4874ba6c867b2f7ff717f6c3f665bd1ae820fbc1e258c1761425: Status 404 returned error can't find the container with id 36cbb1b18dbd4874ba6c867b2f7ff717f6c3f665bd1ae820fbc1e258c1761425 Nov 02 21:45:00.967414 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.965388 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc"] Nov 02 21:45:00.970371 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:00.970343 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm"] Nov 02 21:45:01.003065 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:45:01.003030 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee59c9a7_7469_4f26_af41_95f0bcef64f6.slice/crio-34c1773193bbda86ae3f5449e7271c887e77227c1ab2d1c9d6fb1fb5da2184ca WatchSource:0}: Error finding container 34c1773193bbda86ae3f5449e7271c887e77227c1ab2d1c9d6fb1fb5da2184ca: Status 404 returned error can't find the container with id 34c1773193bbda86ae3f5449e7271c887e77227c1ab2d1c9d6fb1fb5da2184ca Nov 02 21:45:01.168397 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:01.168299 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc" event={"ID":"be71f888-c746-460e-96a2-70a060697231","Type":"ContainerStarted","Data":"405bf4da76a7fb60490ec0a1b67a510382d1197d7aa02ee27646b9e071309602"} Nov 02 21:45:01.169955 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:01.169924 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc" event={"ID":"f454e9fe-c1a8-4057-8afe-329450b2b037","Type":"ContainerStarted","Data":"36cbb1b18dbd4874ba6c867b2f7ff717f6c3f665bd1ae820fbc1e258c1761425"} Nov 02 21:45:01.171437 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:01.171408 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm" event={"ID":"ee59c9a7-7469-4f26-af41-95f0bcef64f6","Type":"ContainerStarted","Data":"34c1773193bbda86ae3f5449e7271c887e77227c1ab2d1c9d6fb1fb5da2184ca"} Nov 02 21:45:01.458321 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:01.458282 4935 patch_prober.go:29] interesting pod/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:45:01.458700 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:01.458335 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" podUID="ff38001c-9b28-473a-bdc7-9ebc8514fe51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:45:02.180745 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:02.180657 4935 generic.go:356] "Generic (PLEG): container finished" podID="f454e9fe-c1a8-4057-8afe-329450b2b037" containerID="fbe4c756cd435eabdbd3f6ab44957ca190dbf938fc3aad3221d82c68460b1723" exitCode=60 Nov 02 21:45:02.180911 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:02.180748 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc" event={"ID":"f454e9fe-c1a8-4057-8afe-329450b2b037","Type":"ContainerDied","Data":"fbe4c756cd435eabdbd3f6ab44957ca190dbf938fc3aad3221d82c68460b1723"} Nov 02 21:45:02.181125 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:02.181095 4935 scope.go:122] "RemoveContainer" containerID="fbe4c756cd435eabdbd3f6ab44957ca190dbf938fc3aad3221d82c68460b1723" Nov 02 21:45:02.183111 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:02.183088 4935 generic.go:356] "Generic (PLEG): container finished" podID="ee59c9a7-7469-4f26-af41-95f0bcef64f6" containerID="a52a4db2e7e279141a48dec45927119687ec4bc95bb5c301195217a3005d996d" exitCode=60 Nov 02 21:45:02.183310 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:02.183293 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm" event={"ID":"ee59c9a7-7469-4f26-af41-95f0bcef64f6","Type":"ContainerDied","Data":"a52a4db2e7e279141a48dec45927119687ec4bc95bb5c301195217a3005d996d"} Nov 02 21:45:02.183567 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:02.183557 4935 scope.go:122] "RemoveContainer" containerID="a52a4db2e7e279141a48dec45927119687ec4bc95bb5c301195217a3005d996d" Nov 02 21:45:02.186406 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:02.185967 4935 generic.go:356] "Generic (PLEG): container finished" podID="be71f888-c746-460e-96a2-70a060697231" containerID="df647b045c0b2520a8f9bd44829f1b1097b530fb46f5092899f1e5dcd8beded4" exitCode=60 Nov 02 21:45:02.186406 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:02.186027 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc" event={"ID":"be71f888-c746-460e-96a2-70a060697231","Type":"ContainerDied","Data":"df647b045c0b2520a8f9bd44829f1b1097b530fb46f5092899f1e5dcd8beded4"} Nov 02 21:45:02.186406 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:02.186266 4935 scope.go:122] "RemoveContainer" containerID="df647b045c0b2520a8f9bd44829f1b1097b530fb46f5092899f1e5dcd8beded4" Nov 02 21:45:03.196510 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:03.196470 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:03.206200 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:03.206166 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc" event={"ID":"be71f888-c746-460e-96a2-70a060697231","Type":"ContainerStarted","Data":"00c3da6c5b1f8a22d5b74d084342b723bbc27d420f56808f53138fef619ad66b"} Nov 02 21:45:03.207974 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:03.207947 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc" event={"ID":"f454e9fe-c1a8-4057-8afe-329450b2b037","Type":"ContainerStarted","Data":"082e2a0f185ad430401f32f09426f0949ffaff1fbb092693c3598b6d7e5b20fe"} Nov 02 21:45:03.210703 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:03.210673 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm" event={"ID":"ee59c9a7-7469-4f26-af41-95f0bcef64f6","Type":"ContainerStarted","Data":"a9ecc7f47a3260f2d7674796d01f8aa0f62f208a59f235e71eb6587ec097ebad"} Nov 02 21:45:03.314002 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:03.313951 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc" podStartSLOduration=3.3139365290000002 podStartE2EDuration="3.313936529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:03.312261224 +0000 UTC m=+2195.751835947" watchObservedRunningTime="2026-11-02 21:45:03.313936529 +0000 UTC m=+2195.753511251" Nov 02 21:45:03.420963 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:03.420910 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc" podStartSLOduration=3.420895449 podStartE2EDuration="3.420895449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:03.41779098 +0000 UTC m=+2195.857365703" watchObservedRunningTime="2026-11-02 21:45:03.420895449 +0000 UTC m=+2195.860470171" Nov 02 21:45:03.525950 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:03.525886 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm" podStartSLOduration=3.525865454 podStartE2EDuration="3.525865454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:03.523299015 +0000 UTC m=+2195.962873738" watchObservedRunningTime="2026-11-02 21:45:03.525865454 +0000 UTC m=+2195.965440177" Nov 02 21:45:04.219585 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:04.219550 4935 generic.go:356] "Generic (PLEG): container finished" podID="ff38001c-9b28-473a-bdc7-9ebc8514fe51" containerID="b6a1e2d518e2203d438d319e1c2cf68f7fb32bcf5500cb73b061397e98464da1" exitCode=0 Nov 02 21:45:04.220029 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:04.219628 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" event={"ID":"ff38001c-9b28-473a-bdc7-9ebc8514fe51","Type":"ContainerDied","Data":"b6a1e2d518e2203d438d319e1c2cf68f7fb32bcf5500cb73b061397e98464da1"} Nov 02 21:45:04.220097 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:04.220081 4935 scope.go:122] "RemoveContainer" containerID="b6a1e2d518e2203d438d319e1c2cf68f7fb32bcf5500cb73b061397e98464da1" Nov 02 21:45:04.221884 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:04.221862 4935 generic.go:356] "Generic (PLEG): container finished" podID="be71f888-c746-460e-96a2-70a060697231" containerID="00c3da6c5b1f8a22d5b74d084342b723bbc27d420f56808f53138fef619ad66b" exitCode=60 Nov 02 21:45:04.221996 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:04.221886 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc" event={"ID":"be71f888-c746-460e-96a2-70a060697231","Type":"ContainerDied","Data":"00c3da6c5b1f8a22d5b74d084342b723bbc27d420f56808f53138fef619ad66b"} Nov 02 21:45:04.221996 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:04.221931 4935 scope.go:122] "RemoveContainer" containerID="df647b045c0b2520a8f9bd44829f1b1097b530fb46f5092899f1e5dcd8beded4" Nov 02 21:45:04.221996 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:04.221860 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:04.222223 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:04.222021 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:04.222223 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:04.222067 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:04.222331 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:04.222219 4935 scope.go:122] "RemoveContainer" containerID="00c3da6c5b1f8a22d5b74d084342b723bbc27d420f56808f53138fef619ad66b" Nov 02 21:45:04.222541 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:45:04.222516 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc_odh-ai-gateway-infra(be71f888-c746-460e-96a2-70a060697231)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc" podUID="be71f888-c746-460e-96a2-70a060697231" Nov 02 21:45:04.224583 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:04.224559 4935 generic.go:356] "Generic (PLEG): container finished" podID="f454e9fe-c1a8-4057-8afe-329450b2b037" containerID="082e2a0f185ad430401f32f09426f0949ffaff1fbb092693c3598b6d7e5b20fe" exitCode=60 Nov 02 21:45:04.224696 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:04.224665 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc" event={"ID":"f454e9fe-c1a8-4057-8afe-329450b2b037","Type":"ContainerDied","Data":"082e2a0f185ad430401f32f09426f0949ffaff1fbb092693c3598b6d7e5b20fe"} Nov 02 21:45:04.224873 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:04.224858 4935 scope.go:122] "RemoveContainer" containerID="082e2a0f185ad430401f32f09426f0949ffaff1fbb092693c3598b6d7e5b20fe" Nov 02 21:45:04.225344 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:45:04.225171 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc_odh-ai-gateway-infra(f454e9fe-c1a8-4057-8afe-329450b2b037)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc" podUID="f454e9fe-c1a8-4057-8afe-329450b2b037" Nov 02 21:45:04.226978 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:04.226959 4935 generic.go:356] "Generic (PLEG): container finished" podID="ee59c9a7-7469-4f26-af41-95f0bcef64f6" containerID="a9ecc7f47a3260f2d7674796d01f8aa0f62f208a59f235e71eb6587ec097ebad" exitCode=60 Nov 02 21:45:04.227069 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:04.227002 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm" event={"ID":"ee59c9a7-7469-4f26-af41-95f0bcef64f6","Type":"ContainerDied","Data":"a9ecc7f47a3260f2d7674796d01f8aa0f62f208a59f235e71eb6587ec097ebad"} Nov 02 21:45:04.227296 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:04.227278 4935 scope.go:122] "RemoveContainer" containerID="a9ecc7f47a3260f2d7674796d01f8aa0f62f208a59f235e71eb6587ec097ebad" Nov 02 21:45:04.227580 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:45:04.227558 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm_odh-ai-gateway-infra(ee59c9a7-7469-4f26-af41-95f0bcef64f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm" podUID="ee59c9a7-7469-4f26-af41-95f0bcef64f6" Nov 02 21:45:04.238163 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:04.238111 4935 scope.go:122] "RemoveContainer" containerID="fbe4c756cd435eabdbd3f6ab44957ca190dbf938fc3aad3221d82c68460b1723" Nov 02 21:45:04.259256 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:04.259233 4935 scope.go:122] "RemoveContainer" containerID="a52a4db2e7e279141a48dec45927119687ec4bc95bb5c301195217a3005d996d" Nov 02 21:45:05.260682 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:05.260650 4935 scope.go:122] "RemoveContainer" containerID="a9ecc7f47a3260f2d7674796d01f8aa0f62f208a59f235e71eb6587ec097ebad" Nov 02 21:45:05.261634 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:45:05.261593 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm_odh-ai-gateway-infra(ee59c9a7-7469-4f26-af41-95f0bcef64f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm" podUID="ee59c9a7-7469-4f26-af41-95f0bcef64f6" Nov 02 21:45:05.262921 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:05.262891 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" event={"ID":"ff38001c-9b28-473a-bdc7-9ebc8514fe51","Type":"ContainerStarted","Data":"c688fa441e4adb8094d6490780966f6b6e1cc6cd1dd26d1ef6784ee9356dee2c"} Nov 02 21:45:05.264980 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:05.264958 4935 scope.go:122] "RemoveContainer" containerID="00c3da6c5b1f8a22d5b74d084342b723bbc27d420f56808f53138fef619ad66b" Nov 02 21:45:05.265293 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:45:05.265253 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc_odh-ai-gateway-infra(be71f888-c746-460e-96a2-70a060697231)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc" podUID="be71f888-c746-460e-96a2-70a060697231" Nov 02 21:45:05.266729 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:05.266712 4935 scope.go:122] "RemoveContainer" containerID="082e2a0f185ad430401f32f09426f0949ffaff1fbb092693c3598b6d7e5b20fe" Nov 02 21:45:05.266948 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:45:05.266929 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc_odh-ai-gateway-infra(f454e9fe-c1a8-4057-8afe-329450b2b037)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc" podUID="f454e9fe-c1a8-4057-8afe-329450b2b037" Nov 02 21:45:05.542401 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:05.541936 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:45:05.542401 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:05.542074 4935 patch_prober.go:29] interesting pod/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 02 21:45:05.542401 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:05.542122 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" podUID="ff38001c-9b28-473a-bdc7-9ebc8514fe51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 02 21:45:06.545155 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:06.544555 4935 patch_prober.go:29] interesting pod/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 02 21:45:06.545155 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:06.544620 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" podUID="ff38001c-9b28-473a-bdc7-9ebc8514fe51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 02 21:45:07.548579 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:07.548539 4935 patch_prober.go:29] interesting pod/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 02 21:45:07.549095 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:07.548607 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" podUID="ff38001c-9b28-473a-bdc7-9ebc8514fe51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 02 21:45:08.044156 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:08.040745 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:08.549191 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:08.546553 4935 patch_prober.go:29] interesting pod/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 02 21:45:08.549191 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:08.546620 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" podUID="ff38001c-9b28-473a-bdc7-9ebc8514fe51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 02 21:45:08.568112 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:08.568044 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:08.778801 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:08.778766 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:09.176938 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:09.176901 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf"] Nov 02 21:45:09.177161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:09.177115 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf" podUID="6616a037-8eba-4c91-9a4e-a1980773f2b1" containerName="maas-api" containerID="cri-o://295f2fdcecbebab521d249856b5e049b66cb8aa8997b6b4e6b024ebf0c949ece" gracePeriod=30 Nov 02 21:45:09.370154 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:09.369987 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:09.398104 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:09.398058 4935 generic.go:356] "Generic (PLEG): container finished" podID="6616a037-8eba-4c91-9a4e-a1980773f2b1" containerID="295f2fdcecbebab521d249856b5e049b66cb8aa8997b6b4e6b024ebf0c949ece" exitCode=0 Nov 02 21:45:09.398336 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:09.398255 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf" event={"ID":"6616a037-8eba-4c91-9a4e-a1980773f2b1","Type":"ContainerDied","Data":"295f2fdcecbebab521d249856b5e049b66cb8aa8997b6b4e6b024ebf0c949ece"} Nov 02 21:45:09.568154 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:09.567710 4935 patch_prober.go:29] interesting pod/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 02 21:45:09.568154 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:09.567769 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" podUID="ff38001c-9b28-473a-bdc7-9ebc8514fe51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 02 21:45:09.669176 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:09.669112 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf" Nov 02 21:45:09.727330 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:09.703547 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6616a037-8eba-4c91-9a4e-a1980773f2b1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6616a037-8eba-4c91-9a4e-a1980773f2b1-maas-api-tls\") pod \"6616a037-8eba-4c91-9a4e-a1980773f2b1\" (UID: \"6616a037-8eba-4c91-9a4e-a1980773f2b1\") " Nov 02 21:45:09.727330 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:09.703676 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6616a037-8eba-4c91-9a4e-a1980773f2b1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6616a037-8eba-4c91-9a4e-a1980773f2b1-metrics-certs\") pod \"6616a037-8eba-4c91-9a4e-a1980773f2b1\" (UID: \"6616a037-8eba-4c91-9a4e-a1980773f2b1\") " Nov 02 21:45:09.727330 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:09.703733 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6616a037-8eba-4c91-9a4e-a1980773f2b1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6616a037-8eba-4c91-9a4e-a1980773f2b1-openshift-service-ca\") pod \"6616a037-8eba-4c91-9a4e-a1980773f2b1\" (UID: \"6616a037-8eba-4c91-9a4e-a1980773f2b1\") " Nov 02 21:45:09.727330 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:09.703932 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6616a037-8eba-4c91-9a4e-a1980773f2b1-kube-api-access-4plzt\" (UniqueName: \"kubernetes.io/projected/6616a037-8eba-4c91-9a4e-a1980773f2b1-kube-api-access-4plzt\") pod \"6616a037-8eba-4c91-9a4e-a1980773f2b1\" (UID: \"6616a037-8eba-4c91-9a4e-a1980773f2b1\") " Nov 02 21:45:09.727330 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:09.714480 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6616a037-8eba-4c91-9a4e-a1980773f2b1-kube-api-access-4plzt" pod "6616a037-8eba-4c91-9a4e-a1980773f2b1" (UID: "6616a037-8eba-4c91-9a4e-a1980773f2b1"). InnerVolumeSpecName "kube-api-access-4plzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:09.727330 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:09.717307 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6616a037-8eba-4c91-9a4e-a1980773f2b1-maas-api-tls" pod "6616a037-8eba-4c91-9a4e-a1980773f2b1" (UID: "6616a037-8eba-4c91-9a4e-a1980773f2b1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:45:09.727330 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:09.719575 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6616a037-8eba-4c91-9a4e-a1980773f2b1-metrics-certs" pod "6616a037-8eba-4c91-9a4e-a1980773f2b1" (UID: "6616a037-8eba-4c91-9a4e-a1980773f2b1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:45:09.740597 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:09.733422 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6616a037-8eba-4c91-9a4e-a1980773f2b1-openshift-service-ca" pod "6616a037-8eba-4c91-9a4e-a1980773f2b1" (UID: "6616a037-8eba-4c91-9a4e-a1980773f2b1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:09.805391 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:09.805361 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4plzt\" (UniqueName: \"kubernetes.io/projected/6616a037-8eba-4c91-9a4e-a1980773f2b1-kube-api-access-4plzt\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:09.805391 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:09.805389 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6616a037-8eba-4c91-9a4e-a1980773f2b1-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:09.805619 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:09.805401 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6616a037-8eba-4c91-9a4e-a1980773f2b1-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:09.805619 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:09.805410 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6616a037-8eba-4c91-9a4e-a1980773f2b1-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:10.056878 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:10.056837 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:10.332819 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:10.332525 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:10.474110 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:10.474061 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf" event={"ID":"6616a037-8eba-4c91-9a4e-a1980773f2b1","Type":"ContainerDied","Data":"6eab084bf66915806c97f7efb8101963afe4cac249951c8110b33033805b0007"} Nov 02 21:45:10.474397 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:10.474365 4935 scope.go:122] "RemoveContainer" containerID="295f2fdcecbebab521d249856b5e049b66cb8aa8997b6b4e6b024ebf0c949ece" Nov 02 21:45:10.474732 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:10.474712 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf" Nov 02 21:45:10.541320 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:10.541280 4935 patch_prober.go:29] interesting pod/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 02 21:45:10.541502 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:10.541346 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" podUID="ff38001c-9b28-473a-bdc7-9ebc8514fe51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 02 21:45:10.628919 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:10.628832 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf"] Nov 02 21:45:10.639466 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:10.639433 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d3617-59866c97cb-7rzbf"] Nov 02 21:45:11.023553 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:11.023482 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:11.230351 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:11.230285 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:11.546651 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:11.546618 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:45:11.547102 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:11.547080 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:45:11.548104 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:11.548076 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:45:11.577374 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:11.577341 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:11.779807 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:11.779766 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6616a037-8eba-4c91-9a4e-a1980773f2b1" path="/var/lib/kubelet/pods/6616a037-8eba-4c91-9a4e-a1980773f2b1/volumes" Nov 02 21:45:12.545644 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:12.545602 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:13.030385 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:13.030341 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:13.777766 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:13.777737 4935 scope.go:122] "RemoveContainer" containerID="a9ecc7f47a3260f2d7674796d01f8aa0f62f208a59f235e71eb6587ec097ebad" Nov 02 21:45:14.493991 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:14.493891 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:14.504370 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:14.504339 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm" event={"ID":"ee59c9a7-7469-4f26-af41-95f0bcef64f6","Type":"ContainerStarted","Data":"9c75af9ed059abbf66b65729aca6efcc84826ab7c2ccd562f9a065b2739c9b3e"} Nov 02 21:45:15.171279 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:15.169368 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:15.513976 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:15.513944 4935 generic.go:356] "Generic (PLEG): container finished" podID="ee59c9a7-7469-4f26-af41-95f0bcef64f6" containerID="9c75af9ed059abbf66b65729aca6efcc84826ab7c2ccd562f9a065b2739c9b3e" exitCode=60 Nov 02 21:45:15.514500 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:15.514024 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm" event={"ID":"ee59c9a7-7469-4f26-af41-95f0bcef64f6","Type":"ContainerDied","Data":"9c75af9ed059abbf66b65729aca6efcc84826ab7c2ccd562f9a065b2739c9b3e"} Nov 02 21:45:15.514500 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:15.514105 4935 scope.go:122] "RemoveContainer" containerID="a9ecc7f47a3260f2d7674796d01f8aa0f62f208a59f235e71eb6587ec097ebad" Nov 02 21:45:15.514605 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:15.514500 4935 scope.go:122] "RemoveContainer" containerID="9c75af9ed059abbf66b65729aca6efcc84826ab7c2ccd562f9a065b2739c9b3e" Nov 02 21:45:15.514845 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:45:15.514825 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm_odh-ai-gateway-infra(ee59c9a7-7469-4f26-af41-95f0bcef64f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm" podUID="ee59c9a7-7469-4f26-af41-95f0bcef64f6" Nov 02 21:45:15.622951 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:15.622917 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm"] Nov 02 21:45:15.672141 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:15.672107 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:15.905163 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:15.905055 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:16.676653 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:16.676622 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm" Nov 02 21:45:16.770102 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:16.770063 4935 scope.go:122] "RemoveContainer" containerID="00c3da6c5b1f8a22d5b74d084342b723bbc27d420f56808f53138fef619ad66b" Nov 02 21:45:16.770311 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:16.770191 4935 scope.go:122] "RemoveContainer" containerID="082e2a0f185ad430401f32f09426f0949ffaff1fbb092693c3598b6d7e5b20fe" Nov 02 21:45:16.809974 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:16.809943 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee59c9a7-7469-4f26-af41-95f0bcef64f6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee59c9a7-7469-4f26-af41-95f0bcef64f6-maas-api-service-ca\") pod \"ee59c9a7-7469-4f26-af41-95f0bcef64f6\" (UID: \"ee59c9a7-7469-4f26-af41-95f0bcef64f6\") " Nov 02 21:45:16.811955 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:16.810165 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee59c9a7-7469-4f26-af41-95f0bcef64f6-kube-api-access-vrppr\" (UniqueName: \"kubernetes.io/projected/ee59c9a7-7469-4f26-af41-95f0bcef64f6-kube-api-access-vrppr\") pod \"ee59c9a7-7469-4f26-af41-95f0bcef64f6\" (UID: \"ee59c9a7-7469-4f26-af41-95f0bcef64f6\") " Nov 02 21:45:16.811955 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:16.810517 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee59c9a7-7469-4f26-af41-95f0bcef64f6-maas-api-service-ca" pod "ee59c9a7-7469-4f26-af41-95f0bcef64f6" (UID: "ee59c9a7-7469-4f26-af41-95f0bcef64f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:16.811955 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:16.810763 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee59c9a7-7469-4f26-af41-95f0bcef64f6-maas-api-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:16.813050 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:16.813023 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee59c9a7-7469-4f26-af41-95f0bcef64f6-kube-api-access-vrppr" pod "ee59c9a7-7469-4f26-af41-95f0bcef64f6" (UID: "ee59c9a7-7469-4f26-af41-95f0bcef64f6"). InnerVolumeSpecName "kube-api-access-vrppr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:16.912759 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:16.912525 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrppr\" (UniqueName: \"kubernetes.io/projected/ee59c9a7-7469-4f26-af41-95f0bcef64f6-kube-api-access-vrppr\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:17.117474 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:17.117438 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:17.355211 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:17.355022 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:17.537248 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:17.537205 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc" event={"ID":"f454e9fe-c1a8-4057-8afe-329450b2b037","Type":"ContainerStarted","Data":"04eca7682fa3959a0bcb260291c018a53776c668240edb141c8c5d7aac012be9"} Nov 02 21:45:17.544727 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:17.539968 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm" event={"ID":"ee59c9a7-7469-4f26-af41-95f0bcef64f6","Type":"ContainerDied","Data":"34c1773193bbda86ae3f5449e7271c887e77227c1ab2d1c9d6fb1fb5da2184ca"} Nov 02 21:45:17.544727 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:17.540027 4935 scope.go:122] "RemoveContainer" containerID="9c75af9ed059abbf66b65729aca6efcc84826ab7c2ccd562f9a065b2739c9b3e" Nov 02 21:45:17.544727 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:17.540149 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm" Nov 02 21:45:17.559559 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:17.559525 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc" event={"ID":"be71f888-c746-460e-96a2-70a060697231","Type":"ContainerStarted","Data":"aa29aba01de3aa17ec7012ee6ff1f716f1b423306f3075eeb712cfcdde32b22e"} Nov 02 21:45:17.690738 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:17.690634 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:17.800985 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:17.800944 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm"] Nov 02 21:45:17.807272 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:17.807237 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6fa533-29894265-77gtm"] Nov 02 21:45:18.418306 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:18.418274 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:18.567272 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:18.567238 4935 generic.go:356] "Generic (PLEG): container finished" podID="be71f888-c746-460e-96a2-70a060697231" containerID="aa29aba01de3aa17ec7012ee6ff1f716f1b423306f3075eeb712cfcdde32b22e" exitCode=60 Nov 02 21:45:18.567424 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:18.567312 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc" event={"ID":"be71f888-c746-460e-96a2-70a060697231","Type":"ContainerDied","Data":"aa29aba01de3aa17ec7012ee6ff1f716f1b423306f3075eeb712cfcdde32b22e"} Nov 02 21:45:18.567424 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:18.567370 4935 scope.go:122] "RemoveContainer" containerID="00c3da6c5b1f8a22d5b74d084342b723bbc27d420f56808f53138fef619ad66b" Nov 02 21:45:18.567803 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:18.567782 4935 scope.go:122] "RemoveContainer" containerID="aa29aba01de3aa17ec7012ee6ff1f716f1b423306f3075eeb712cfcdde32b22e" Nov 02 21:45:18.568190 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:45:18.568125 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc_odh-ai-gateway-infra(be71f888-c746-460e-96a2-70a060697231)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc" podUID="be71f888-c746-460e-96a2-70a060697231" Nov 02 21:45:18.569517 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:18.569492 4935 generic.go:356] "Generic (PLEG): container finished" podID="f454e9fe-c1a8-4057-8afe-329450b2b037" containerID="04eca7682fa3959a0bcb260291c018a53776c668240edb141c8c5d7aac012be9" exitCode=60 Nov 02 21:45:18.569600 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:18.569567 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc" event={"ID":"f454e9fe-c1a8-4057-8afe-329450b2b037","Type":"ContainerDied","Data":"04eca7682fa3959a0bcb260291c018a53776c668240edb141c8c5d7aac012be9"} Nov 02 21:45:18.569927 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:18.569906 4935 scope.go:122] "RemoveContainer" containerID="04eca7682fa3959a0bcb260291c018a53776c668240edb141c8c5d7aac012be9" Nov 02 21:45:18.570221 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:45:18.570179 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc_odh-ai-gateway-infra(f454e9fe-c1a8-4057-8afe-329450b2b037)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc" podUID="f454e9fe-c1a8-4057-8afe-329450b2b037" Nov 02 21:45:18.581072 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:18.581049 4935 scope.go:122] "RemoveContainer" containerID="082e2a0f185ad430401f32f09426f0949ffaff1fbb092693c3598b6d7e5b20fe" Nov 02 21:45:18.657988 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:18.657957 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc"] Nov 02 21:45:18.887017 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:18.886985 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc"] Nov 02 21:45:19.787773 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:19.787734 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee59c9a7-7469-4f26-af41-95f0bcef64f6" path="/var/lib/kubelet/pods/ee59c9a7-7469-4f26-af41-95f0bcef64f6/volumes" Nov 02 21:45:19.790419 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:19.790396 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc" Nov 02 21:45:19.792994 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:19.792966 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc" Nov 02 21:45:19.857428 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:19.857393 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f454e9fe-c1a8-4057-8afe-329450b2b037-kube-api-access-rc5pm\" (UniqueName: \"kubernetes.io/projected/f454e9fe-c1a8-4057-8afe-329450b2b037-kube-api-access-rc5pm\") pod \"f454e9fe-c1a8-4057-8afe-329450b2b037\" (UID: \"f454e9fe-c1a8-4057-8afe-329450b2b037\") " Nov 02 21:45:19.857428 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:19.857443 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f454e9fe-c1a8-4057-8afe-329450b2b037-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f454e9fe-c1a8-4057-8afe-329450b2b037-maas-api-service-ca\") pod \"f454e9fe-c1a8-4057-8afe-329450b2b037\" (UID: \"f454e9fe-c1a8-4057-8afe-329450b2b037\") " Nov 02 21:45:19.858017 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:19.857883 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f454e9fe-c1a8-4057-8afe-329450b2b037-maas-api-service-ca" pod "f454e9fe-c1a8-4057-8afe-329450b2b037" (UID: "f454e9fe-c1a8-4057-8afe-329450b2b037"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:19.858250 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:19.858228 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f454e9fe-c1a8-4057-8afe-329450b2b037-maas-api-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:19.859728 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:19.859701 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f454e9fe-c1a8-4057-8afe-329450b2b037-kube-api-access-rc5pm" pod "f454e9fe-c1a8-4057-8afe-329450b2b037" (UID: "f454e9fe-c1a8-4057-8afe-329450b2b037"). InnerVolumeSpecName "kube-api-access-rc5pm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:19.959124 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:19.959089 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be71f888-c746-460e-96a2-70a060697231-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be71f888-c746-460e-96a2-70a060697231-maas-api-service-ca\") pod \"be71f888-c746-460e-96a2-70a060697231\" (UID: \"be71f888-c746-460e-96a2-70a060697231\") " Nov 02 21:45:19.959677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:19.959354 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be71f888-c746-460e-96a2-70a060697231-kube-api-access-hxxd7\" (UniqueName: \"kubernetes.io/projected/be71f888-c746-460e-96a2-70a060697231-kube-api-access-hxxd7\") pod \"be71f888-c746-460e-96a2-70a060697231\" (UID: \"be71f888-c746-460e-96a2-70a060697231\") " Nov 02 21:45:19.959677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:19.959437 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be71f888-c746-460e-96a2-70a060697231-maas-api-service-ca" pod "be71f888-c746-460e-96a2-70a060697231" (UID: "be71f888-c746-460e-96a2-70a060697231"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:19.959830 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:19.959691 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be71f888-c746-460e-96a2-70a060697231-maas-api-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:19.959830 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:19.959705 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc5pm\" (UniqueName: \"kubernetes.io/projected/f454e9fe-c1a8-4057-8afe-329450b2b037-kube-api-access-rc5pm\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:19.961569 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:19.961541 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be71f888-c746-460e-96a2-70a060697231-kube-api-access-hxxd7" pod "be71f888-c746-460e-96a2-70a060697231" (UID: "be71f888-c746-460e-96a2-70a060697231"). InnerVolumeSpecName "kube-api-access-hxxd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:20.060823 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:20.060742 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxxd7\" (UniqueName: \"kubernetes.io/projected/be71f888-c746-460e-96a2-70a060697231-kube-api-access-hxxd7\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:20.093385 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:20.093348 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:20.308292 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:20.308260 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:20.588843 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:20.588812 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc" Nov 02 21:45:20.589043 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:20.588844 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc" event={"ID":"be71f888-c746-460e-96a2-70a060697231","Type":"ContainerDied","Data":"405bf4da76a7fb60490ec0a1b67a510382d1197d7aa02ee27646b9e071309602"} Nov 02 21:45:20.589043 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:20.588906 4935 scope.go:122] "RemoveContainer" containerID="aa29aba01de3aa17ec7012ee6ff1f716f1b423306f3075eeb712cfcdde32b22e" Nov 02 21:45:20.590578 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:20.590555 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc" Nov 02 21:45:20.590695 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:20.590627 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc" event={"ID":"f454e9fe-c1a8-4057-8afe-329450b2b037","Type":"ContainerDied","Data":"36cbb1b18dbd4874ba6c867b2f7ff717f6c3f665bd1ae820fbc1e258c1761425"} Nov 02 21:45:20.600380 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:20.600355 4935 scope.go:122] "RemoveContainer" containerID="04eca7682fa3959a0bcb260291c018a53776c668240edb141c8c5d7aac012be9" Nov 02 21:45:20.744422 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:20.744384 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc"] Nov 02 21:45:20.790552 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:20.790517 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-387f88-29894265-t5zqc"] Nov 02 21:45:20.894631 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:20.894544 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc"] Nov 02 21:45:20.900154 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:20.900108 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d6cb38-29894265-kdvjc"] Nov 02 21:45:21.778518 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:21.778481 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be71f888-c746-460e-96a2-70a060697231" path="/var/lib/kubelet/pods/be71f888-c746-460e-96a2-70a060697231/volumes" Nov 02 21:45:21.779221 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:21.779199 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f454e9fe-c1a8-4057-8afe-329450b2b037" path="/var/lib/kubelet/pods/f454e9fe-c1a8-4057-8afe-329450b2b037/volumes" Nov 02 21:45:22.844003 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:22.843966 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:23.271583 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:23.271543 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:23.488918 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:23.488883 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:24.448909 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:24.448869 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:28.468419 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:45:28.468381 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:45:29.850606 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:29.850562 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:30.524582 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:30.524539 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:37.225860 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:37.225813 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:37.881556 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:37.881510 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:38.093948 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:38.093905 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:39.150880 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:39.150840 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:44.284551 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:44.284507 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:47.415768 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:47.415726 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:47.642865 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:47.642831 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:48.934486 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:48.934443 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:49.421384 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:49.421345 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:49.708077 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:49.707978 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:50.879759 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:50.879719 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:51.092936 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:51.092894 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:51.496287 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:51.496239 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:51.726774 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:51.726724 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:51.936763 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:51.936729 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:52.151677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:52.151595 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:54.346498 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:54.346450 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:55.852217 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:55.852181 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:56.073242 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:56.073181 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:56.534475 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:56.534439 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:56.755248 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:56.755185 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:57.479610 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:57.479575 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:45:58.471805 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:45:58.471760 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:45:59.453351 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:59.453313 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq"] Nov 02 21:45:59.453828 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:45:59.453582 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" podUID="ee0a0727-6bb6-4f61-82cd-80e292cbf912" containerName="istio-proxy" containerID="cri-o://8be6618945007b5ec8a6ba35a9b10765e46bec8b259c909a3f6e5cfa75718c4a" gracePeriod=30 Nov 02 21:46:00.164505 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:00.164472 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:00.389777 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:00.389736 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:01.547291 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:01.547256 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:46:01.547681 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:01.547329 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" podUID="ee0a0727-6bb6-4f61-82cd-80e292cbf912" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:46:02.834339 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:02.834298 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85"] Nov 02 21:46:02.834831 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:02.834620 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" podUID="cd7972de-f06d-4db9-84ae-15dcce006b32" containerName="istio-proxy" containerID="cri-o://7ac550da95e04b036089f66afd368b6448c8a95db48ff3c791e3a4dd929262b1" gracePeriod=30 Nov 02 21:46:03.034462 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:03.034419 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:04.485945 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.485912 4935 patch_prober.go:29] interesting pod/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:46:04.486296 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.485990 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" podUID="cd7972de-f06d-4db9-84ae-15dcce006b32" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:46:04.625810 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.625679 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:04.793746 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.793716 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:46:04.866099 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.866061 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:04.875390 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.875360 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-credential-socket\") pod \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " Nov 02 21:46:04.875565 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.875425 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istio-token\" (UniqueName: \"kubernetes.io/projected/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istio-token\") pod \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " Nov 02 21:46:04.875565 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.875459 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istio-podinfo\") pod \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " Nov 02 21:46:04.875565 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.875512 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istio-envoy\") pod \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " Nov 02 21:46:04.875731 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.875670 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istiod-ca-cert\") pod \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " Nov 02 21:46:04.875731 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.875682 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-credential-socket" pod "ee0a0727-6bb6-4f61-82cd-80e292cbf912" (UID: "ee0a0727-6bb6-4f61-82cd-80e292cbf912"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:04.875731 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.875707 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-workload-certs\") pod \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " Nov 02 21:46:04.875882 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.875791 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee0a0727-6bb6-4f61-82cd-80e292cbf912-kube-api-access-swbzk\" (UniqueName: \"kubernetes.io/projected/ee0a0727-6bb6-4f61-82cd-80e292cbf912-kube-api-access-swbzk\") pod \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " Nov 02 21:46:04.875882 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.875847 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istio-data\") pod \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " Nov 02 21:46:04.875990 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.875910 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-workload-socket\") pod \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\" (UID: \"ee0a0727-6bb6-4f61-82cd-80e292cbf912\") " Nov 02 21:46:04.876417 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.876338 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istio-data" pod "ee0a0727-6bb6-4f61-82cd-80e292cbf912" (UID: "ee0a0727-6bb6-4f61-82cd-80e292cbf912"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:04.876417 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.876394 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istiod-ca-cert" pod "ee0a0727-6bb6-4f61-82cd-80e292cbf912" (UID: "ee0a0727-6bb6-4f61-82cd-80e292cbf912"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:46:04.876568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.876427 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-workload-certs" pod "ee0a0727-6bb6-4f61-82cd-80e292cbf912" (UID: "ee0a0727-6bb6-4f61-82cd-80e292cbf912"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:04.876568 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.876442 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:04.876635 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.876584 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-workload-socket" pod "ee0a0727-6bb6-4f61-82cd-80e292cbf912" (UID: "ee0a0727-6bb6-4f61-82cd-80e292cbf912"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:04.879193 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.879118 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istio-podinfo" pod "ee0a0727-6bb6-4f61-82cd-80e292cbf912" (UID: "ee0a0727-6bb6-4f61-82cd-80e292cbf912"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:46:04.879397 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.879372 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istio-envoy" pod "ee0a0727-6bb6-4f61-82cd-80e292cbf912" (UID: "ee0a0727-6bb6-4f61-82cd-80e292cbf912"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:04.879514 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.879466 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istio-token" pod "ee0a0727-6bb6-4f61-82cd-80e292cbf912" (UID: "ee0a0727-6bb6-4f61-82cd-80e292cbf912"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:46:04.880206 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.880180 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee0a0727-6bb6-4f61-82cd-80e292cbf912-kube-api-access-swbzk" pod "ee0a0727-6bb6-4f61-82cd-80e292cbf912" (UID: "ee0a0727-6bb6-4f61-82cd-80e292cbf912"). InnerVolumeSpecName "kube-api-access-swbzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:46:04.921896 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.921864 4935 generic.go:356] "Generic (PLEG): container finished" podID="ee0a0727-6bb6-4f61-82cd-80e292cbf912" containerID="8be6618945007b5ec8a6ba35a9b10765e46bec8b259c909a3f6e5cfa75718c4a" exitCode=0 Nov 02 21:46:04.922064 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.921919 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" Nov 02 21:46:04.922064 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.921954 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" event={"ID":"ee0a0727-6bb6-4f61-82cd-80e292cbf912","Type":"ContainerDied","Data":"8be6618945007b5ec8a6ba35a9b10765e46bec8b259c909a3f6e5cfa75718c4a"} Nov 02 21:46:04.922064 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.922011 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq" event={"ID":"ee0a0727-6bb6-4f61-82cd-80e292cbf912","Type":"ContainerDied","Data":"059b5dbe24fe2e6361a5e7a97da50d0d25809cbcf07dd0ba513f09a9285ba5df"} Nov 02 21:46:04.922064 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.922035 4935 scope.go:122] "RemoveContainer" containerID="8be6618945007b5ec8a6ba35a9b10765e46bec8b259c909a3f6e5cfa75718c4a" Nov 02 21:46:04.934915 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.934890 4935 scope.go:122] "RemoveContainer" containerID="8be6618945007b5ec8a6ba35a9b10765e46bec8b259c909a3f6e5cfa75718c4a" Nov 02 21:46:04.935239 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:46:04.935214 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8be6618945007b5ec8a6ba35a9b10765e46bec8b259c909a3f6e5cfa75718c4a\": container with ID starting with 8be6618945007b5ec8a6ba35a9b10765e46bec8b259c909a3f6e5cfa75718c4a not found: ID does not exist" containerID="8be6618945007b5ec8a6ba35a9b10765e46bec8b259c909a3f6e5cfa75718c4a" Nov 02 21:46:04.935338 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.935246 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8be6618945007b5ec8a6ba35a9b10765e46bec8b259c909a3f6e5cfa75718c4a"} err="failed to get container status \"8be6618945007b5ec8a6ba35a9b10765e46bec8b259c909a3f6e5cfa75718c4a\": rpc error: code = NotFound desc = could not find container \"8be6618945007b5ec8a6ba35a9b10765e46bec8b259c909a3f6e5cfa75718c4a\": container with ID starting with 8be6618945007b5ec8a6ba35a9b10765e46bec8b259c909a3f6e5cfa75718c4a not found: ID does not exist" Nov 02 21:46:04.977858 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.977819 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:04.977858 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.977852 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:04.978078 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.977870 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swbzk\" (UniqueName: \"kubernetes.io/projected/ee0a0727-6bb6-4f61-82cd-80e292cbf912-kube-api-access-swbzk\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:04.978078 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.977887 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:04.978078 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.977901 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:04.978078 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.977918 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:04.978078 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.977931 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:04.978078 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:04.977944 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee0a0727-6bb6-4f61-82cd-80e292cbf912-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:05.053369 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:05.053231 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq"] Nov 02 21:46:05.060188 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:05.060160 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-dj9zq"] Nov 02 21:46:05.777933 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:05.777898 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee0a0727-6bb6-4f61-82cd-80e292cbf912" path="/var/lib/kubelet/pods/ee0a0727-6bb6-4f61-82cd-80e292cbf912/volumes" Nov 02 21:46:08.104186 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.104032 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:46:08.114615 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.114462 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:08.223328 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.223293 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-credential-socket\") pod \"cd7972de-f06d-4db9-84ae-15dcce006b32\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " Nov 02 21:46:08.223328 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.223329 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-workload-socket\") pod \"cd7972de-f06d-4db9-84ae-15dcce006b32\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " Nov 02 21:46:08.223530 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.223465 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-istio-envoy\") pod \"cd7972de-f06d-4db9-84ae-15dcce006b32\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " Nov 02 21:46:08.223530 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.223502 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd7972de-f06d-4db9-84ae-15dcce006b32-kube-api-access-zx6fv\" (UniqueName: \"kubernetes.io/projected/cd7972de-f06d-4db9-84ae-15dcce006b32-kube-api-access-zx6fv\") pod \"cd7972de-f06d-4db9-84ae-15dcce006b32\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " Nov 02 21:46:08.223623 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.223530 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd7972de-f06d-4db9-84ae-15dcce006b32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd7972de-f06d-4db9-84ae-15dcce006b32-istiod-ca-cert\") pod \"cd7972de-f06d-4db9-84ae-15dcce006b32\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " Nov 02 21:46:08.223623 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.223565 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-credential-socket" pod "cd7972de-f06d-4db9-84ae-15dcce006b32" (UID: "cd7972de-f06d-4db9-84ae-15dcce006b32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:08.223623 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.223586 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd7972de-f06d-4db9-84ae-15dcce006b32-istio-token\" (UniqueName: \"kubernetes.io/projected/cd7972de-f06d-4db9-84ae-15dcce006b32-istio-token\") pod \"cd7972de-f06d-4db9-84ae-15dcce006b32\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " Nov 02 21:46:08.223764 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.223652 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-workload-certs\") pod \"cd7972de-f06d-4db9-84ae-15dcce006b32\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " Nov 02 21:46:08.223764 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.223693 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd7972de-f06d-4db9-84ae-15dcce006b32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd7972de-f06d-4db9-84ae-15dcce006b32-istio-podinfo\") pod \"cd7972de-f06d-4db9-84ae-15dcce006b32\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " Nov 02 21:46:08.223764 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.223710 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-workload-socket" pod "cd7972de-f06d-4db9-84ae-15dcce006b32" (UID: "cd7972de-f06d-4db9-84ae-15dcce006b32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:08.223764 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.223726 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-istio-data\") pod \"cd7972de-f06d-4db9-84ae-15dcce006b32\" (UID: \"cd7972de-f06d-4db9-84ae-15dcce006b32\") " Nov 02 21:46:08.223986 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.223940 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-workload-certs" pod "cd7972de-f06d-4db9-84ae-15dcce006b32" (UID: "cd7972de-f06d-4db9-84ae-15dcce006b32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:08.224057 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.223978 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd7972de-f06d-4db9-84ae-15dcce006b32-istiod-ca-cert" pod "cd7972de-f06d-4db9-84ae-15dcce006b32" (UID: "cd7972de-f06d-4db9-84ae-15dcce006b32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:46:08.224190 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.224165 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-istio-data" pod "cd7972de-f06d-4db9-84ae-15dcce006b32" (UID: "cd7972de-f06d-4db9-84ae-15dcce006b32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:08.224364 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.224343 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd7972de-f06d-4db9-84ae-15dcce006b32-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:08.224430 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.224372 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:08.224430 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.224386 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:08.224430 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.224400 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:08.224430 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.224414 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:08.225745 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.225724 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-istio-envoy" pod "cd7972de-f06d-4db9-84ae-15dcce006b32" (UID: "cd7972de-f06d-4db9-84ae-15dcce006b32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:08.225895 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.225868 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd7972de-f06d-4db9-84ae-15dcce006b32-istio-token" pod "cd7972de-f06d-4db9-84ae-15dcce006b32" (UID: "cd7972de-f06d-4db9-84ae-15dcce006b32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:46:08.225995 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.225950 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd7972de-f06d-4db9-84ae-15dcce006b32-kube-api-access-zx6fv" pod "cd7972de-f06d-4db9-84ae-15dcce006b32" (UID: "cd7972de-f06d-4db9-84ae-15dcce006b32"). InnerVolumeSpecName "kube-api-access-zx6fv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:46:08.226281 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.226257 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd7972de-f06d-4db9-84ae-15dcce006b32-istio-podinfo" pod "cd7972de-f06d-4db9-84ae-15dcce006b32" (UID: "cd7972de-f06d-4db9-84ae-15dcce006b32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:46:08.325527 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.325490 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd7972de-f06d-4db9-84ae-15dcce006b32-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:08.325527 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.325521 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx6fv\" (UniqueName: \"kubernetes.io/projected/cd7972de-f06d-4db9-84ae-15dcce006b32-kube-api-access-zx6fv\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:08.325527 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.325533 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd7972de-f06d-4db9-84ae-15dcce006b32-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:08.325768 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.325542 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd7972de-f06d-4db9-84ae-15dcce006b32-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:08.951953 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.951915 4935 generic.go:356] "Generic (PLEG): container finished" podID="cd7972de-f06d-4db9-84ae-15dcce006b32" containerID="7ac550da95e04b036089f66afd368b6448c8a95db48ff3c791e3a4dd929262b1" exitCode=0 Nov 02 21:46:08.952147 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.951983 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" Nov 02 21:46:08.952147 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.952003 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" event={"ID":"cd7972de-f06d-4db9-84ae-15dcce006b32","Type":"ContainerDied","Data":"7ac550da95e04b036089f66afd368b6448c8a95db48ff3c791e3a4dd929262b1"} Nov 02 21:46:08.952147 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.952072 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85" event={"ID":"cd7972de-f06d-4db9-84ae-15dcce006b32","Type":"ContainerDied","Data":"72fb433d82f950132512e0a2532222217650ddb14102f359f55ab4a7f36f02d9"} Nov 02 21:46:08.952147 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.952105 4935 scope.go:122] "RemoveContainer" containerID="7ac550da95e04b036089f66afd368b6448c8a95db48ff3c791e3a4dd929262b1" Nov 02 21:46:08.962708 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.962686 4935 scope.go:122] "RemoveContainer" containerID="7ac550da95e04b036089f66afd368b6448c8a95db48ff3c791e3a4dd929262b1" Nov 02 21:46:08.963018 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:46:08.962996 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ac550da95e04b036089f66afd368b6448c8a95db48ff3c791e3a4dd929262b1\": container with ID starting with 7ac550da95e04b036089f66afd368b6448c8a95db48ff3c791e3a4dd929262b1 not found: ID does not exist" containerID="7ac550da95e04b036089f66afd368b6448c8a95db48ff3c791e3a4dd929262b1" Nov 02 21:46:08.963168 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:08.963114 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ac550da95e04b036089f66afd368b6448c8a95db48ff3c791e3a4dd929262b1"} err="failed to get container status \"7ac550da95e04b036089f66afd368b6448c8a95db48ff3c791e3a4dd929262b1\": rpc error: code = NotFound desc = could not find container \"7ac550da95e04b036089f66afd368b6448c8a95db48ff3c791e3a4dd929262b1\": container with ID starting with 7ac550da95e04b036089f66afd368b6448c8a95db48ff3c791e3a4dd929262b1 not found: ID does not exist" Nov 02 21:46:09.106402 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:09.106363 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85"] Nov 02 21:46:09.152663 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:09.152626 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f765a-openshift-default-7df7757c5f-t7l85"] Nov 02 21:46:09.777046 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:09.777010 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd7972de-f06d-4db9-84ae-15dcce006b32" path="/var/lib/kubelet/pods/cd7972de-f06d-4db9-84ae-15dcce006b32/volumes" Nov 02 21:46:13.123000 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:13.122966 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:18.264829 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:18.264790 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:23.330003 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:23.329958 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:25.088925 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:25.088889 4935 generic.go:356] "Generic (PLEG): container finished" podID="cf9d00da-108f-4cd1-8ca8-5f89ba59c624" containerID="e2f2517e6c5b39eed75d461e5d6f4bf0ee4fa50480678707a15766b101019baf" exitCode=0 Nov 02 21:46:25.089392 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:25.088966 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" event={"ID":"cf9d00da-108f-4cd1-8ca8-5f89ba59c624","Type":"ContainerDied","Data":"e2f2517e6c5b39eed75d461e5d6f4bf0ee4fa50480678707a15766b101019baf"} Nov 02 21:46:25.089392 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:25.089364 4935 scope.go:122] "RemoveContainer" containerID="e2f2517e6c5b39eed75d461e5d6f4bf0ee4fa50480678707a15766b101019baf" Nov 02 21:46:25.090816 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:25.090787 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:46:25.091035 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:25.090850 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:46:25.091035 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:25.090884 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:46:26.092996 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:26.092963 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:26.097915 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:26.097883 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" event={"ID":"cf9d00da-108f-4cd1-8ca8-5f89ba59c624","Type":"ContainerStarted","Data":"1aaf70280cdebab112ecdae1e00e0f04bed5b20576660a87a38b527d0e281c30"} Nov 02 21:46:26.747463 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:26.747427 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:46:26.749540 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:26.749507 4935 patch_prober.go:29] interesting pod/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 21:46:26.749698 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:26.749571 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" podUID="cf9d00da-108f-4cd1-8ca8-5f89ba59c624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 21:46:27.015071 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:27.015036 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:46:27.748535 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:27.748495 4935 patch_prober.go:29] interesting pod/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 21:46:27.748903 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:27.748581 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" podUID="cf9d00da-108f-4cd1-8ca8-5f89ba59c624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 21:46:28.264813 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:28.264776 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:28.479302 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:28.479271 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:28.489731 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:46:28.489681 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:46:28.700620 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:28.700517 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:28.748153 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:28.747475 4935 patch_prober.go:29] interesting pod/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 21:46:28.748153 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:28.747536 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" podUID="cf9d00da-108f-4cd1-8ca8-5f89ba59c624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 21:46:29.751156 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:29.750507 4935 patch_prober.go:29] interesting pod/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 21:46:29.751156 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:29.750581 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" podUID="cf9d00da-108f-4cd1-8ca8-5f89ba59c624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 21:46:30.171155 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:30.167990 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:30.366695 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:30.366652 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:30.747557 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:30.747518 4935 patch_prober.go:29] interesting pod/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 21:46:30.747864 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:30.747834 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" podUID="cf9d00da-108f-4cd1-8ca8-5f89ba59c624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 21:46:30.981249 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:30.981214 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf"] Nov 02 21:46:31.143950 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:31.143895 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:31.746340 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:31.746305 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:46:31.747346 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:31.747320 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:46:33.653466 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:33.653432 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:40.519719 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:40.519632 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:46.001754 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.001710 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" podUID="6676be94-69f0-477e-9423-d45d58eb0d35" containerName="main" containerID="cri-o://da883ae9608c3eaade8491008a575e4dbc53a99c2b96cfb0033a8ee36c6c1cf7" gracePeriod=45 Nov 02 21:46:46.297012 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.296980 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" Nov 02 21:46:46.308360 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.308324 4935 generic.go:356] "Generic (PLEG): container finished" podID="6676be94-69f0-477e-9423-d45d58eb0d35" containerID="da883ae9608c3eaade8491008a575e4dbc53a99c2b96cfb0033a8ee36c6c1cf7" exitCode=0 Nov 02 21:46:46.308538 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.308364 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" Nov 02 21:46:46.308538 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.308408 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" event={"ID":"6676be94-69f0-477e-9423-d45d58eb0d35","Type":"ContainerDied","Data":"da883ae9608c3eaade8491008a575e4dbc53a99c2b96cfb0033a8ee36c6c1cf7"} Nov 02 21:46:46.308538 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.308458 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf" event={"ID":"6676be94-69f0-477e-9423-d45d58eb0d35","Type":"ContainerDied","Data":"2e26feb29f413c53d91153ae366142774d73a2feb2f70bb6840eba36346e07b7"} Nov 02 21:46:46.308538 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.308482 4935 scope.go:122] "RemoveContainer" containerID="da883ae9608c3eaade8491008a575e4dbc53a99c2b96cfb0033a8ee36c6c1cf7" Nov 02 21:46:46.321845 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.321827 4935 scope.go:122] "RemoveContainer" containerID="da883ae9608c3eaade8491008a575e4dbc53a99c2b96cfb0033a8ee36c6c1cf7" Nov 02 21:46:46.322142 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:46:46.322113 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da883ae9608c3eaade8491008a575e4dbc53a99c2b96cfb0033a8ee36c6c1cf7\": container with ID starting with da883ae9608c3eaade8491008a575e4dbc53a99c2b96cfb0033a8ee36c6c1cf7 not found: ID does not exist" containerID="da883ae9608c3eaade8491008a575e4dbc53a99c2b96cfb0033a8ee36c6c1cf7" Nov 02 21:46:46.322202 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.322151 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da883ae9608c3eaade8491008a575e4dbc53a99c2b96cfb0033a8ee36c6c1cf7"} err="failed to get container status \"da883ae9608c3eaade8491008a575e4dbc53a99c2b96cfb0033a8ee36c6c1cf7\": rpc error: code = NotFound desc = could not find container \"da883ae9608c3eaade8491008a575e4dbc53a99c2b96cfb0033a8ee36c6c1cf7\": container with ID starting with da883ae9608c3eaade8491008a575e4dbc53a99c2b96cfb0033a8ee36c6c1cf7 not found: ID does not exist" Nov 02 21:46:46.421518 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.421477 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6676be94-69f0-477e-9423-d45d58eb0d35-tls-certs\" (UniqueName: \"kubernetes.io/secret/6676be94-69f0-477e-9423-d45d58eb0d35-tls-certs\") pod \"6676be94-69f0-477e-9423-d45d58eb0d35\" (UID: \"6676be94-69f0-477e-9423-d45d58eb0d35\") " Nov 02 21:46:46.421719 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.421632 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6676be94-69f0-477e-9423-d45d58eb0d35-kube-api-access-hwn8j\" (UniqueName: \"kubernetes.io/projected/6676be94-69f0-477e-9423-d45d58eb0d35-kube-api-access-hwn8j\") pod \"6676be94-69f0-477e-9423-d45d58eb0d35\" (UID: \"6676be94-69f0-477e-9423-d45d58eb0d35\") " Nov 02 21:46:46.421719 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.421677 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6676be94-69f0-477e-9423-d45d58eb0d35-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6676be94-69f0-477e-9423-d45d58eb0d35-model-cache\") pod \"6676be94-69f0-477e-9423-d45d58eb0d35\" (UID: \"6676be94-69f0-477e-9423-d45d58eb0d35\") " Nov 02 21:46:46.421849 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.421739 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6676be94-69f0-477e-9423-d45d58eb0d35-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6676be94-69f0-477e-9423-d45d58eb0d35-tmp-dir\") pod \"6676be94-69f0-477e-9423-d45d58eb0d35\" (UID: \"6676be94-69f0-477e-9423-d45d58eb0d35\") " Nov 02 21:46:46.421849 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.421819 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6676be94-69f0-477e-9423-d45d58eb0d35-home\" (UniqueName: \"kubernetes.io/empty-dir/6676be94-69f0-477e-9423-d45d58eb0d35-home\") pod \"6676be94-69f0-477e-9423-d45d58eb0d35\" (UID: \"6676be94-69f0-477e-9423-d45d58eb0d35\") " Nov 02 21:46:46.421964 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.421874 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6676be94-69f0-477e-9423-d45d58eb0d35-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6676be94-69f0-477e-9423-d45d58eb0d35-dshm\") pod \"6676be94-69f0-477e-9423-d45d58eb0d35\" (UID: \"6676be94-69f0-477e-9423-d45d58eb0d35\") " Nov 02 21:46:46.423016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.422987 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6676be94-69f0-477e-9423-d45d58eb0d35-model-cache" pod "6676be94-69f0-477e-9423-d45d58eb0d35" (UID: "6676be94-69f0-477e-9423-d45d58eb0d35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:46.423411 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.423379 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6676be94-69f0-477e-9423-d45d58eb0d35-tmp-dir" pod "6676be94-69f0-477e-9423-d45d58eb0d35" (UID: "6676be94-69f0-477e-9423-d45d58eb0d35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:46.423554 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.423531 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6676be94-69f0-477e-9423-d45d58eb0d35-home" pod "6676be94-69f0-477e-9423-d45d58eb0d35" (UID: "6676be94-69f0-477e-9423-d45d58eb0d35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:46.434113 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.431148 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6676be94-69f0-477e-9423-d45d58eb0d35-dshm" pod "6676be94-69f0-477e-9423-d45d58eb0d35" (UID: "6676be94-69f0-477e-9423-d45d58eb0d35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:46.441148 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.434265 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6676be94-69f0-477e-9423-d45d58eb0d35-tls-certs" pod "6676be94-69f0-477e-9423-d45d58eb0d35" (UID: "6676be94-69f0-477e-9423-d45d58eb0d35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:46:46.441904 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.441866 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6676be94-69f0-477e-9423-d45d58eb0d35-kube-api-access-hwn8j" pod "6676be94-69f0-477e-9423-d45d58eb0d35" (UID: "6676be94-69f0-477e-9423-d45d58eb0d35"). InnerVolumeSpecName "kube-api-access-hwn8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:46:46.524221 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.524115 4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6676be94-69f0-477e-9423-d45d58eb0d35-home\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:46.524221 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.524176 4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6676be94-69f0-477e-9423-d45d58eb0d35-dshm\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:46.524221 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.524190 4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6676be94-69f0-477e-9423-d45d58eb0d35-tls-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:46.524221 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.524206 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwn8j\" (UniqueName: \"kubernetes.io/projected/6676be94-69f0-477e-9423-d45d58eb0d35-kube-api-access-hwn8j\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:46.524221 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.524222 4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6676be94-69f0-477e-9423-d45d58eb0d35-model-cache\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:46.524504 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.524265 4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6676be94-69f0-477e-9423-d45d58eb0d35-tmp-dir\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:46.724344 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.724300 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf"] Nov 02 21:46:46.729879 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:46.729848 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-e12f99/ipp-route-e12f99-kserve-75844fbb5c-7q5jf"] Nov 02 21:46:47.256551 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:47.256516 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:47.778008 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:47.777973 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6676be94-69f0-477e-9423-d45d58eb0d35" path="/var/lib/kubelet/pods/6676be94-69f0-477e-9423-d45d58eb0d35/volumes" Nov 02 21:46:52.474910 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:52.474881 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:46:58.472643 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:46:58.472597 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:46:59.236955 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:46:59.236920 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:04.167321 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:04.167233 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:04.362759 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:04.362714 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:04.606597 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:04.606559 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:07.310388 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:07.310327 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:08.322667 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:08.322625 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:08.560730 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:08.560685 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:08.763276 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:08.763249 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:08.965067 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:08.965027 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:09.396980 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.396941 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr"] Nov 02 21:47:09.397508 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397444 4935 cpu_manager.go:414] "containerMap: removing container" podUID="be71f888-c746-460e-96a2-70a060697231" containerName="cleanup" Nov 02 21:47:09.397508 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397460 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be71f888-c746-460e-96a2-70a060697231" containerName="cleanup" Nov 02 21:47:09.397508 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397480 4935 cpu_manager.go:414] "containerMap: removing container" podUID="6616a037-8eba-4c91-9a4e-a1980773f2b1" containerName="maas-api" Nov 02 21:47:09.397508 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397486 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6616a037-8eba-4c91-9a4e-a1980773f2b1" containerName="maas-api" Nov 02 21:47:09.397508 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397495 4935 cpu_manager.go:414] "containerMap: removing container" podUID="f454e9fe-c1a8-4057-8afe-329450b2b037" containerName="cleanup" Nov 02 21:47:09.397508 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397500 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f454e9fe-c1a8-4057-8afe-329450b2b037" containerName="cleanup" Nov 02 21:47:09.397508 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397508 4935 cpu_manager.go:414] "containerMap: removing container" podUID="ee0a0727-6bb6-4f61-82cd-80e292cbf912" containerName="istio-proxy" Nov 02 21:47:09.397508 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397512 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee0a0727-6bb6-4f61-82cd-80e292cbf912" containerName="istio-proxy" Nov 02 21:47:09.397928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397520 4935 cpu_manager.go:414] "containerMap: removing container" podUID="f454e9fe-c1a8-4057-8afe-329450b2b037" containerName="cleanup" Nov 02 21:47:09.397928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397524 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f454e9fe-c1a8-4057-8afe-329450b2b037" containerName="cleanup" Nov 02 21:47:09.397928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397536 4935 cpu_manager.go:414] "containerMap: removing container" podUID="6676be94-69f0-477e-9423-d45d58eb0d35" containerName="main" Nov 02 21:47:09.397928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397540 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6676be94-69f0-477e-9423-d45d58eb0d35" containerName="main" Nov 02 21:47:09.397928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397552 4935 cpu_manager.go:414] "containerMap: removing container" podUID="f454e9fe-c1a8-4057-8afe-329450b2b037" containerName="cleanup" Nov 02 21:47:09.397928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397556 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f454e9fe-c1a8-4057-8afe-329450b2b037" containerName="cleanup" Nov 02 21:47:09.397928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397563 4935 cpu_manager.go:414] "containerMap: removing container" podUID="ee59c9a7-7469-4f26-af41-95f0bcef64f6" containerName="cleanup" Nov 02 21:47:09.397928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397567 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee59c9a7-7469-4f26-af41-95f0bcef64f6" containerName="cleanup" Nov 02 21:47:09.397928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397574 4935 cpu_manager.go:414] "containerMap: removing container" podUID="ee59c9a7-7469-4f26-af41-95f0bcef64f6" containerName="cleanup" Nov 02 21:47:09.397928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397578 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee59c9a7-7469-4f26-af41-95f0bcef64f6" containerName="cleanup" Nov 02 21:47:09.397928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397596 4935 cpu_manager.go:414] "containerMap: removing container" podUID="cd7972de-f06d-4db9-84ae-15dcce006b32" containerName="istio-proxy" Nov 02 21:47:09.397928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397602 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd7972de-f06d-4db9-84ae-15dcce006b32" containerName="istio-proxy" Nov 02 21:47:09.397928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397613 4935 cpu_manager.go:414] "containerMap: removing container" podUID="be71f888-c746-460e-96a2-70a060697231" containerName="cleanup" Nov 02 21:47:09.397928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397617 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be71f888-c746-460e-96a2-70a060697231" containerName="cleanup" Nov 02 21:47:09.397928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397688 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee0a0727-6bb6-4f61-82cd-80e292cbf912" containerName="istio-proxy" Nov 02 21:47:09.397928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397697 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="f454e9fe-c1a8-4057-8afe-329450b2b037" containerName="cleanup" Nov 02 21:47:09.397928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397704 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="6676be94-69f0-477e-9423-d45d58eb0d35" containerName="main" Nov 02 21:47:09.397928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397710 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="f454e9fe-c1a8-4057-8afe-329450b2b037" containerName="cleanup" Nov 02 21:47:09.397928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397715 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee59c9a7-7469-4f26-af41-95f0bcef64f6" containerName="cleanup" Nov 02 21:47:09.397928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397720 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee59c9a7-7469-4f26-af41-95f0bcef64f6" containerName="cleanup" Nov 02 21:47:09.397928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397727 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="be71f888-c746-460e-96a2-70a060697231" containerName="cleanup" Nov 02 21:47:09.397928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397732 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd7972de-f06d-4db9-84ae-15dcce006b32" containerName="istio-proxy" Nov 02 21:47:09.397928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397738 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="be71f888-c746-460e-96a2-70a060697231" containerName="cleanup" Nov 02 21:47:09.397928 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.397746 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="6616a037-8eba-4c91-9a4e-a1980773f2b1" containerName="maas-api" Nov 02 21:47:09.401280 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.401254 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr" Nov 02 21:47:09.503179 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.503123 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-cc0d6a-serving-cert\"" Nov 02 21:47:09.506291 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.506259 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr"] Nov 02 21:47:09.513748 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.513722 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr\" (UID: \"d82a4ce3-ed44-4aa7-ad58-4c4a0233234b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr" Nov 02 21:47:09.513907 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.513793 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr\" (UID: \"d82a4ce3-ed44-4aa7-ad58-4c4a0233234b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr" Nov 02 21:47:09.514181 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.514157 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-metrics-certs\") pod \"maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr\" (UID: \"d82a4ce3-ed44-4aa7-ad58-4c4a0233234b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr" Nov 02 21:47:09.514277 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.514257 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z7zj\" (UniqueName: \"kubernetes.io/projected/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-kube-api-access-7z7zj\") pod \"maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr\" (UID: \"d82a4ce3-ed44-4aa7-ad58-4c4a0233234b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr" Nov 02 21:47:09.615404 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.615368 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z7zj\" (UniqueName: \"kubernetes.io/projected/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-kube-api-access-7z7zj\") pod \"maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr\" (UID: \"d82a4ce3-ed44-4aa7-ad58-4c4a0233234b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr" Nov 02 21:47:09.615593 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.615447 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr\" (UID: \"d82a4ce3-ed44-4aa7-ad58-4c4a0233234b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr" Nov 02 21:47:09.615593 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.615510 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr\" (UID: \"d82a4ce3-ed44-4aa7-ad58-4c4a0233234b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr" Nov 02 21:47:09.615593 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:47:09.615578 4935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-serving-cert: secret "maas-api-e2e-praxis-b-cc0d6a-serving-cert" not found Nov 02 21:47:09.615792 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.615634 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-metrics-certs\") pod \"maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr\" (UID: \"d82a4ce3-ed44-4aa7-ad58-4c4a0233234b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr" Nov 02 21:47:09.615792 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:47:09.615644 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-maas-api-tls podName:d82a4ce3-ed44-4aa7-ad58-4c4a0233234b nodeName:}" failed. No retries permitted until 2026-11-02 21:47:10.115628543 +0000 UTC m=+2322.555203260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-maas-api-tls") pod "maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr" (UID: "d82a4ce3-ed44-4aa7-ad58-4c4a0233234b") : secret "maas-api-e2e-praxis-b-cc0d6a-serving-cert" not found Nov 02 21:47:09.616191 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.616171 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr\" (UID: \"d82a4ce3-ed44-4aa7-ad58-4c4a0233234b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr" Nov 02 21:47:09.617988 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.617965 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-metrics-certs\") pod \"maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr\" (UID: \"d82a4ce3-ed44-4aa7-ad58-4c4a0233234b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr" Nov 02 21:47:09.673927 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.673851 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z7zj\" (UniqueName: \"kubernetes.io/projected/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-kube-api-access-7z7zj\") pod \"maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr\" (UID: \"d82a4ce3-ed44-4aa7-ad58-4c4a0233234b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr" Nov 02 21:47:09.712250 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.712193 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:09.975718 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:09.975632 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:10.120891 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:10.120849 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr\" (UID: \"d82a4ce3-ed44-4aa7-ad58-4c4a0233234b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr" Nov 02 21:47:10.123321 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:10.123286 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr\" (UID: \"d82a4ce3-ed44-4aa7-ad58-4c4a0233234b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr" Nov 02 21:47:10.206103 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:10.206055 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:10.320161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:10.320096 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr" Nov 02 21:47:10.699361 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:10.698551 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr"] Nov 02 21:47:10.701145 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:47:10.701100 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd82a4ce3_ed44_4aa7_ad58_4c4a0233234b.slice/crio-d92e5295b353294bf7356333bcca16ddcf06f53debabc1248b67b6b75169a287 WatchSource:0}: Error finding container d92e5295b353294bf7356333bcca16ddcf06f53debabc1248b67b6b75169a287: Status 404 returned error can't find the container with id d92e5295b353294bf7356333bcca16ddcf06f53debabc1248b67b6b75169a287 Nov 02 21:47:11.485650 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:11.485611 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr" event={"ID":"d82a4ce3-ed44-4aa7-ad58-4c4a0233234b","Type":"ContainerStarted","Data":"d92e5295b353294bf7356333bcca16ddcf06f53debabc1248b67b6b75169a287"} Nov 02 21:47:12.494096 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:12.494052 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr" event={"ID":"d82a4ce3-ed44-4aa7-ad58-4c4a0233234b","Type":"ContainerStarted","Data":"099a72541ac8765554802311279fd53b75d1e256521c974b29c9c492988021d8"} Nov 02 21:47:12.494592 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:12.494230 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr" Nov 02 21:47:12.603893 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:12.603840 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr" podStartSLOduration=2.734954504 podStartE2EDuration="3.603825205s" totalImagesPullingTime="868.870701ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:47:12.60011647 +0000 UTC m=+2325.039691197" watchObservedRunningTime="2026-11-02 21:47:12.603825205 +0000 UTC m=+2325.043399928" Nov 02 21:47:14.697379 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:14.697345 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:14.961478 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:14.961396 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:15.404584 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:15.404553 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:18.506319 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:18.506286 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr" Nov 02 21:47:20.142284 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:20.142238 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:20.856205 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:20.856166 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:25.824306 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:25.824253 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:28.494315 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:47:28.494181 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:47:30.490693 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:30.490654 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:30.861617 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:30.861584 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:35.869628 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:35.869590 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:40.844693 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:40.844640 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:44.886192 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:44.886160 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:45.523260 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:45.523230 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:45.777170 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:45.777062 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:47.904511 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:47.904471 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:50.217363 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:50.217317 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:50.790439 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:50.790402 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:51.008696 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:51.008658 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:51.588427 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:51.588390 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:52.203360 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:52.203327 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:52.428881 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:52.428845 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:53.543796 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:53.543766 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:53.763505 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:53.763476 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:54.541830 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:54.541798 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:54.769076 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:54.769037 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:55.033730 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:55.033650 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:55.279781 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:55.279743 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:55.509092 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:55.509016 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:56.281120 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:56.281084 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:56.918445 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:56.918411 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:57.806098 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:57.806061 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:58.286552 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:58.286495 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:58.474390 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:47:58.474351 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:47:58.973306 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:58.973260 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:59.407880 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:59.407848 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:59.633385 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:59.633343 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:47:59.860830 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:47:59.860794 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:00.727725 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:00.727693 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:00.952016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:00.951978 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:02.251872 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:02.251829 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:02.666118 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:02.666073 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:02.919318 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:02.919226 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:04.673505 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:04.673467 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:06.901121 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:06.901088 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:08.410006 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:08.409929 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:10.889558 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:10.889519 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:11.393966 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:11.393929 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:14.533055 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:14.533019 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:20.062791 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:20.062750 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:23.984924 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:23.984889 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:25.118184 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:25.118143 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:25.595154 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:25.594418 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:25.941515 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:25.941417 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:26.297489 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:26.297452 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:28.241256 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:28.241223 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 21:48:28.250218 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:28.250195 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 21:48:28.251609 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:28.251586 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 21:48:28.259734 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:28.259715 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 21:48:28.482865 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:48:28.482816 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:48:29.299860 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:29.299819 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:29.985944 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:29.985904 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:30.213818 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:30.213781 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:30.829496 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:30.829462 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:31.392807 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:31.392775 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:32.301289 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:32.301258 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:33.137251 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:33.137191 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:33.367841 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:33.367776 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:36.699818 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:36.699769 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:38.111339 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:38.111308 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:38.848027 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:38.847987 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:39.062036 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:39.061992 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:41.961835 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:41.961798 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:43.705932 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:43.705901 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:47.447317 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:47.447251 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:52.925917 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:52.925867 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:57.923557 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:48:57.923524 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:48:58.469802 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:48:58.469766 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:49:02.130539 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:02.130495 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd"] Nov 02 21:49:02.131049 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:02.130760 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd" podUID="a273e83e-21f3-4f31-babb-94579b007a61" containerName="maas-api" containerID="cri-o://815a51ec5d841f5e51cb131effb3fb5673913a5729e333971a30da21494afdeb" gracePeriod=30 Nov 02 21:49:02.324869 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:02.324834 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:02.446063 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:02.446032 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd" Nov 02 21:49:02.527155 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:02.525875 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a273e83e-21f3-4f31-babb-94579b007a61-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a273e83e-21f3-4f31-babb-94579b007a61-openshift-service-ca\") pod \"a273e83e-21f3-4f31-babb-94579b007a61\" (UID: \"a273e83e-21f3-4f31-babb-94579b007a61\") " Nov 02 21:49:02.527155 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:02.526056 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a273e83e-21f3-4f31-babb-94579b007a61-kube-api-access-qxlfd\" (UniqueName: \"kubernetes.io/projected/a273e83e-21f3-4f31-babb-94579b007a61-kube-api-access-qxlfd\") pod \"a273e83e-21f3-4f31-babb-94579b007a61\" (UID: \"a273e83e-21f3-4f31-babb-94579b007a61\") " Nov 02 21:49:02.527155 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:02.526098 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a273e83e-21f3-4f31-babb-94579b007a61-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a273e83e-21f3-4f31-babb-94579b007a61-maas-api-tls\") pod \"a273e83e-21f3-4f31-babb-94579b007a61\" (UID: \"a273e83e-21f3-4f31-babb-94579b007a61\") " Nov 02 21:49:02.527155 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:02.526242 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a273e83e-21f3-4f31-babb-94579b007a61-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a273e83e-21f3-4f31-babb-94579b007a61-metrics-certs\") pod \"a273e83e-21f3-4f31-babb-94579b007a61\" (UID: \"a273e83e-21f3-4f31-babb-94579b007a61\") " Nov 02 21:49:02.529335 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:02.529271 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a273e83e-21f3-4f31-babb-94579b007a61-openshift-service-ca" pod "a273e83e-21f3-4f31-babb-94579b007a61" (UID: "a273e83e-21f3-4f31-babb-94579b007a61"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:49:02.554510 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:02.554437 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a273e83e-21f3-4f31-babb-94579b007a61-metrics-certs" pod "a273e83e-21f3-4f31-babb-94579b007a61" (UID: "a273e83e-21f3-4f31-babb-94579b007a61"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:49:02.587156 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:02.583771 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a273e83e-21f3-4f31-babb-94579b007a61-maas-api-tls" pod "a273e83e-21f3-4f31-babb-94579b007a61" (UID: "a273e83e-21f3-4f31-babb-94579b007a61"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:49:02.595153 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:02.591742 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a273e83e-21f3-4f31-babb-94579b007a61-kube-api-access-qxlfd" pod "a273e83e-21f3-4f31-babb-94579b007a61" (UID: "a273e83e-21f3-4f31-babb-94579b007a61"). InnerVolumeSpecName "kube-api-access-qxlfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:02.628475 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:02.628435 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxlfd\" (UniqueName: \"kubernetes.io/projected/a273e83e-21f3-4f31-babb-94579b007a61-kube-api-access-qxlfd\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:02.628475 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:02.628465 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a273e83e-21f3-4f31-babb-94579b007a61-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:02.628475 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:02.628475 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a273e83e-21f3-4f31-babb-94579b007a61-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:02.628475 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:02.628484 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a273e83e-21f3-4f31-babb-94579b007a61-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:03.238680 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.238647 4935 generic.go:356] "Generic (PLEG): container finished" podID="a273e83e-21f3-4f31-babb-94579b007a61" containerID="815a51ec5d841f5e51cb131effb3fb5673913a5729e333971a30da21494afdeb" exitCode=0 Nov 02 21:49:03.239121 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.238694 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd" event={"ID":"a273e83e-21f3-4f31-babb-94579b007a61","Type":"ContainerDied","Data":"815a51ec5d841f5e51cb131effb3fb5673913a5729e333971a30da21494afdeb"} Nov 02 21:49:03.239121 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.238749 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd" event={"ID":"a273e83e-21f3-4f31-babb-94579b007a61","Type":"ContainerDied","Data":"90556fc66239a46af90fe0e1bfd83d18c4a4815050dc271aafbb1fc5125f0f42"} Nov 02 21:49:03.239121 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.238772 4935 scope.go:122] "RemoveContainer" containerID="815a51ec5d841f5e51cb131effb3fb5673913a5729e333971a30da21494afdeb" Nov 02 21:49:03.239121 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.238698 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd" Nov 02 21:49:03.249485 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.249466 4935 scope.go:122] "RemoveContainer" containerID="815a51ec5d841f5e51cb131effb3fb5673913a5729e333971a30da21494afdeb" Nov 02 21:49:03.249761 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:49:03.249743 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"815a51ec5d841f5e51cb131effb3fb5673913a5729e333971a30da21494afdeb\": container with ID starting with 815a51ec5d841f5e51cb131effb3fb5673913a5729e333971a30da21494afdeb not found: ID does not exist" containerID="815a51ec5d841f5e51cb131effb3fb5673913a5729e333971a30da21494afdeb" Nov 02 21:49:03.249816 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.249770 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815a51ec5d841f5e51cb131effb3fb5673913a5729e333971a30da21494afdeb"} err="failed to get container status \"815a51ec5d841f5e51cb131effb3fb5673913a5729e333971a30da21494afdeb\": rpc error: code = NotFound desc = could not find container \"815a51ec5d841f5e51cb131effb3fb5673913a5729e333971a30da21494afdeb\": container with ID starting with 815a51ec5d841f5e51cb131effb3fb5673913a5729e333971a30da21494afdeb not found: ID does not exist" Nov 02 21:49:03.394973 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.394935 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd"] Nov 02 21:49:03.441386 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.441295 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-be6df0-68586d7645-twjhd"] Nov 02 21:49:03.778114 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.778077 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a273e83e-21f3-4f31-babb-94579b007a61" path="/var/lib/kubelet/pods/a273e83e-21f3-4f31-babb-94579b007a61/volumes" Nov 02 21:49:03.793814 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.793785 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v"] Nov 02 21:49:03.794331 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.794317 4935 cpu_manager.go:414] "containerMap: removing container" podUID="ee59c9a7-7469-4f26-af41-95f0bcef64f6" containerName="cleanup" Nov 02 21:49:03.794392 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.794332 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee59c9a7-7469-4f26-af41-95f0bcef64f6" containerName="cleanup" Nov 02 21:49:03.794392 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.794357 4935 cpu_manager.go:414] "containerMap: removing container" podUID="a273e83e-21f3-4f31-babb-94579b007a61" containerName="maas-api" Nov 02 21:49:03.794392 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.794363 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a273e83e-21f3-4f31-babb-94579b007a61" containerName="maas-api" Nov 02 21:49:03.794392 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.794373 4935 cpu_manager.go:414] "containerMap: removing container" podUID="be71f888-c746-460e-96a2-70a060697231" containerName="cleanup" Nov 02 21:49:03.794392 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.794378 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be71f888-c746-460e-96a2-70a060697231" containerName="cleanup" Nov 02 21:49:03.794566 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.794473 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="f454e9fe-c1a8-4057-8afe-329450b2b037" containerName="cleanup" Nov 02 21:49:03.794566 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.794481 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee59c9a7-7469-4f26-af41-95f0bcef64f6" containerName="cleanup" Nov 02 21:49:03.794566 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.794493 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="a273e83e-21f3-4f31-babb-94579b007a61" containerName="maas-api" Nov 02 21:49:03.794566 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.794498 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="be71f888-c746-460e-96a2-70a060697231" containerName="cleanup" Nov 02 21:49:03.798966 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.798944 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v" Nov 02 21:49:03.844083 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.844056 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8210d255-0b34-442c-b473-87051911a385-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v\" (UID: \"8210d255-0b34-442c-b473-87051911a385\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v" Nov 02 21:49:03.844338 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.844307 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8210d255-0b34-442c-b473-87051911a385-metrics-certs\") pod \"maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v\" (UID: \"8210d255-0b34-442c-b473-87051911a385\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v" Nov 02 21:49:03.844455 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.844354 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zjzw\" (UniqueName: \"kubernetes.io/projected/8210d255-0b34-442c-b473-87051911a385-kube-api-access-4zjzw\") pod \"maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v\" (UID: \"8210d255-0b34-442c-b473-87051911a385\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v" Nov 02 21:49:03.844527 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.844492 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8210d255-0b34-442c-b473-87051911a385-maas-api-tls\") pod \"maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v\" (UID: \"8210d255-0b34-442c-b473-87051911a385\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v" Nov 02 21:49:03.848343 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.848319 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-942164-serving-cert\"" Nov 02 21:49:03.850901 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.850877 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v"] Nov 02 21:49:03.945633 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.945595 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8210d255-0b34-442c-b473-87051911a385-metrics-certs\") pod \"maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v\" (UID: \"8210d255-0b34-442c-b473-87051911a385\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v" Nov 02 21:49:03.945818 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.945649 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zjzw\" (UniqueName: \"kubernetes.io/projected/8210d255-0b34-442c-b473-87051911a385-kube-api-access-4zjzw\") pod \"maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v\" (UID: \"8210d255-0b34-442c-b473-87051911a385\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v" Nov 02 21:49:03.945818 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.945716 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8210d255-0b34-442c-b473-87051911a385-maas-api-tls\") pod \"maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v\" (UID: \"8210d255-0b34-442c-b473-87051911a385\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v" Nov 02 21:49:03.945818 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.945798 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8210d255-0b34-442c-b473-87051911a385-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v\" (UID: \"8210d255-0b34-442c-b473-87051911a385\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v" Nov 02 21:49:03.946078 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:49:03.946055 4935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-serving-cert: secret "maas-api-e2e-praxis-b-942164-serving-cert" not found Nov 02 21:49:03.946176 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:49:03.946121 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8210d255-0b34-442c-b473-87051911a385-maas-api-tls podName:8210d255-0b34-442c-b473-87051911a385 nodeName:}" failed. No retries permitted until 2026-11-02 21:49:04.446101114 +0000 UTC m=+2436.885675816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8210d255-0b34-442c-b473-87051911a385-maas-api-tls") pod "maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v" (UID: "8210d255-0b34-442c-b473-87051911a385") : secret "maas-api-e2e-praxis-b-942164-serving-cert" not found Nov 02 21:49:03.946711 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.946667 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8210d255-0b34-442c-b473-87051911a385-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v\" (UID: \"8210d255-0b34-442c-b473-87051911a385\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v" Nov 02 21:49:03.948622 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.948597 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8210d255-0b34-442c-b473-87051911a385-metrics-certs\") pod \"maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v\" (UID: \"8210d255-0b34-442c-b473-87051911a385\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v" Nov 02 21:49:03.999303 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:03.999275 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zjzw\" (UniqueName: \"kubernetes.io/projected/8210d255-0b34-442c-b473-87051911a385-kube-api-access-4zjzw\") pod \"maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v\" (UID: \"8210d255-0b34-442c-b473-87051911a385\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v" Nov 02 21:49:04.083473 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:04.083386 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:04.305577 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:04.305537 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:04.456799 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:04.456720 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8210d255-0b34-442c-b473-87051911a385-maas-api-tls\") pod \"maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v\" (UID: \"8210d255-0b34-442c-b473-87051911a385\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v" Nov 02 21:49:04.459192 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:04.459162 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8210d255-0b34-442c-b473-87051911a385-maas-api-tls\") pod \"maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v\" (UID: \"8210d255-0b34-442c-b473-87051911a385\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v" Nov 02 21:49:04.712829 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:04.712739 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v" Nov 02 21:49:05.082802 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:05.082777 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v"] Nov 02 21:49:05.084901 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:49:05.084867 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8210d255_0b34_442c_b473_87051911a385.slice/crio-1e0248b54e8666c8d940311d791f0405f8d2bdca8f0a012745af4e8ade6658ba WatchSource:0}: Error finding container 1e0248b54e8666c8d940311d791f0405f8d2bdca8f0a012745af4e8ade6658ba: Status 404 returned error can't find the container with id 1e0248b54e8666c8d940311d791f0405f8d2bdca8f0a012745af4e8ade6658ba Nov 02 21:49:05.253829 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:05.253792 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v" event={"ID":"8210d255-0b34-442c-b473-87051911a385","Type":"ContainerStarted","Data":"1e0248b54e8666c8d940311d791f0405f8d2bdca8f0a012745af4e8ade6658ba"} Nov 02 21:49:06.263020 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:06.262977 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v" event={"ID":"8210d255-0b34-442c-b473-87051911a385","Type":"ContainerStarted","Data":"e86047cd896d589be579b658a2b11e47a940a823be824992ba2de806ba9ec9d9"} Nov 02 21:49:06.263474 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:06.263086 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v" Nov 02 21:49:06.370410 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:06.370360 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v" podStartSLOduration=2.472816464 podStartE2EDuration="3.370345698s" totalImagesPullingTime="897.529234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:49:06.369897959 +0000 UTC m=+2438.809472706" watchObservedRunningTime="2026-11-02 21:49:06.370345698 +0000 UTC m=+2438.809920435" Nov 02 21:49:06.505524 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:06.505486 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:06.734855 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:06.734761 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:08.127662 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:08.127630 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:08.516017 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:08.515974 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k"] Nov 02 21:49:08.516270 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:08.516240 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" podUID="ff38001c-9b28-473a-bdc7-9ebc8514fe51" containerName="istio-proxy" containerID="cri-o://c688fa441e4adb8094d6490780966f6b6e1cc6cd1dd26d1ef6784ee9356dee2c" gracePeriod=30 Nov 02 21:49:08.604724 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:08.604693 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:08.846677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:08.842021 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:09.316766 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:09.316737 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:09.539743 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:09.539707 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:10.112402 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:10.112371 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:10.718962 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:10.718924 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:10.945305 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:10.945270 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:11.550088 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:11.549732 4935 patch_prober.go:29] interesting pod/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:49:11.550088 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:11.549806 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" podUID="ff38001c-9b28-473a-bdc7-9ebc8514fe51" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:49:12.117882 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:12.117851 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:12.275791 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:12.275760 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v" Nov 02 21:49:13.815222 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.815191 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:49:13.870929 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.870896 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-workload-socket\") pod \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " Nov 02 21:49:13.871126 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.870973 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff38001c-9b28-473a-bdc7-9ebc8514fe51-kube-api-access-9vcx8\" (UniqueName: \"kubernetes.io/projected/ff38001c-9b28-473a-bdc7-9ebc8514fe51-kube-api-access-9vcx8\") pod \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " Nov 02 21:49:13.871126 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.871003 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istio-data\") pod \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " Nov 02 21:49:13.871126 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.871028 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-workload-certs\") pod \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " Nov 02 21:49:13.871325 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.871206 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istiod-ca-cert\") pod \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " Nov 02 21:49:13.871325 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.871241 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istio-token\" (UniqueName: \"kubernetes.io/projected/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istio-token\") pod \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " Nov 02 21:49:13.871325 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.871253 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-workload-socket" pod "ff38001c-9b28-473a-bdc7-9ebc8514fe51" (UID: "ff38001c-9b28-473a-bdc7-9ebc8514fe51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:13.871325 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.871304 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-credential-socket\") pod \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " Nov 02 21:49:13.871532 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.871423 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istio-data" pod "ff38001c-9b28-473a-bdc7-9ebc8514fe51" (UID: "ff38001c-9b28-473a-bdc7-9ebc8514fe51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:13.871532 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.871419 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istio-podinfo\") pod \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " Nov 02 21:49:13.871642 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.871601 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istiod-ca-cert" pod "ff38001c-9b28-473a-bdc7-9ebc8514fe51" (UID: "ff38001c-9b28-473a-bdc7-9ebc8514fe51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:49:13.871642 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.871604 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istio-envoy\") pod \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\" (UID: \"ff38001c-9b28-473a-bdc7-9ebc8514fe51\") " Nov 02 21:49:13.872177 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.871827 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-credential-socket" pod "ff38001c-9b28-473a-bdc7-9ebc8514fe51" (UID: "ff38001c-9b28-473a-bdc7-9ebc8514fe51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:13.872177 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.871968 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-workload-certs" pod "ff38001c-9b28-473a-bdc7-9ebc8514fe51" (UID: "ff38001c-9b28-473a-bdc7-9ebc8514fe51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:13.872387 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.872367 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:13.872453 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.872392 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:13.872453 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.872406 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:13.872453 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.872419 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:13.872453 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.872432 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:13.873852 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.873814 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istio-podinfo" pod "ff38001c-9b28-473a-bdc7-9ebc8514fe51" (UID: "ff38001c-9b28-473a-bdc7-9ebc8514fe51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:49:13.873951 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.873881 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff38001c-9b28-473a-bdc7-9ebc8514fe51-kube-api-access-9vcx8" pod "ff38001c-9b28-473a-bdc7-9ebc8514fe51" (UID: "ff38001c-9b28-473a-bdc7-9ebc8514fe51"). InnerVolumeSpecName "kube-api-access-9vcx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:13.873951 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.873911 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istio-token" pod "ff38001c-9b28-473a-bdc7-9ebc8514fe51" (UID: "ff38001c-9b28-473a-bdc7-9ebc8514fe51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:13.874379 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.874359 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istio-envoy" pod "ff38001c-9b28-473a-bdc7-9ebc8514fe51" (UID: "ff38001c-9b28-473a-bdc7-9ebc8514fe51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:13.973223 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.973190 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:13.973223 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.973217 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:13.973223 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.973229 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vcx8\" (UniqueName: \"kubernetes.io/projected/ff38001c-9b28-473a-bdc7-9ebc8514fe51-kube-api-access-9vcx8\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:13.973465 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.973238 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff38001c-9b28-473a-bdc7-9ebc8514fe51-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:13.993826 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:13.993764 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:14.322024 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:14.321987 4935 generic.go:356] "Generic (PLEG): container finished" podID="ff38001c-9b28-473a-bdc7-9ebc8514fe51" containerID="c688fa441e4adb8094d6490780966f6b6e1cc6cd1dd26d1ef6784ee9356dee2c" exitCode=0 Nov 02 21:49:14.322252 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:14.322045 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" Nov 02 21:49:14.322252 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:14.322076 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" event={"ID":"ff38001c-9b28-473a-bdc7-9ebc8514fe51","Type":"ContainerDied","Data":"c688fa441e4adb8094d6490780966f6b6e1cc6cd1dd26d1ef6784ee9356dee2c"} Nov 02 21:49:14.322252 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:14.322181 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k" event={"ID":"ff38001c-9b28-473a-bdc7-9ebc8514fe51","Type":"ContainerDied","Data":"54ee0e4a6a0f02d89f5c3cc89f2e18951cb0182b464981e60bc5fe42d7b7165c"} Nov 02 21:49:14.322252 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:14.322207 4935 scope.go:122] "RemoveContainer" containerID="c688fa441e4adb8094d6490780966f6b6e1cc6cd1dd26d1ef6784ee9356dee2c" Nov 02 21:49:14.333168 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:14.333145 4935 scope.go:122] "RemoveContainer" containerID="b6a1e2d518e2203d438d319e1c2cf68f7fb32bcf5500cb73b061397e98464da1" Nov 02 21:49:14.345713 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:14.345693 4935 scope.go:122] "RemoveContainer" containerID="c688fa441e4adb8094d6490780966f6b6e1cc6cd1dd26d1ef6784ee9356dee2c" Nov 02 21:49:14.346025 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:49:14.346006 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c688fa441e4adb8094d6490780966f6b6e1cc6cd1dd26d1ef6784ee9356dee2c\": container with ID starting with c688fa441e4adb8094d6490780966f6b6e1cc6cd1dd26d1ef6784ee9356dee2c not found: ID does not exist" containerID="c688fa441e4adb8094d6490780966f6b6e1cc6cd1dd26d1ef6784ee9356dee2c" Nov 02 21:49:14.346090 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:14.346033 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c688fa441e4adb8094d6490780966f6b6e1cc6cd1dd26d1ef6784ee9356dee2c"} err="failed to get container status \"c688fa441e4adb8094d6490780966f6b6e1cc6cd1dd26d1ef6784ee9356dee2c\": rpc error: code = NotFound desc = could not find container \"c688fa441e4adb8094d6490780966f6b6e1cc6cd1dd26d1ef6784ee9356dee2c\": container with ID starting with c688fa441e4adb8094d6490780966f6b6e1cc6cd1dd26d1ef6784ee9356dee2c not found: ID does not exist" Nov 02 21:49:14.346090 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:14.346051 4935 scope.go:122] "RemoveContainer" containerID="b6a1e2d518e2203d438d319e1c2cf68f7fb32bcf5500cb73b061397e98464da1" Nov 02 21:49:14.346389 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:49:14.346370 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6a1e2d518e2203d438d319e1c2cf68f7fb32bcf5500cb73b061397e98464da1\": container with ID starting with b6a1e2d518e2203d438d319e1c2cf68f7fb32bcf5500cb73b061397e98464da1 not found: ID does not exist" containerID="b6a1e2d518e2203d438d319e1c2cf68f7fb32bcf5500cb73b061397e98464da1" Nov 02 21:49:14.346432 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:14.346396 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a1e2d518e2203d438d319e1c2cf68f7fb32bcf5500cb73b061397e98464da1"} err="failed to get container status \"b6a1e2d518e2203d438d319e1c2cf68f7fb32bcf5500cb73b061397e98464da1\": rpc error: code = NotFound desc = could not find container \"b6a1e2d518e2203d438d319e1c2cf68f7fb32bcf5500cb73b061397e98464da1\": container with ID starting with b6a1e2d518e2203d438d319e1c2cf68f7fb32bcf5500cb73b061397e98464da1 not found: ID does not exist" Nov 02 21:49:14.481361 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:14.481327 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k"] Nov 02 21:49:14.498181 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:14.498125 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:14.523836 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:14.523808 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-be6df0-openshift-default-7f8bc578d5-9gh2k"] Nov 02 21:49:14.723009 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:14.722931 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:15.776483 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:15.776448 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff38001c-9b28-473a-bdc7-9ebc8514fe51" path="/var/lib/kubelet/pods/ff38001c-9b28-473a-bdc7-9ebc8514fe51/volumes" Nov 02 21:49:16.046221 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:16.046072 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:16.430973 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:16.430888 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4"] Nov 02 21:49:16.431162 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:16.431115 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" podUID="cf9d00da-108f-4cd1-8ca8-5f89ba59c624" containerName="istio-proxy" containerID="cri-o://1aaf70280cdebab112ecdae1e00e0f04bed5b20576660a87a38b527d0e281c30" gracePeriod=30 Nov 02 21:49:16.516361 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:16.516326 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:16.750017 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:16.749937 4935 patch_prober.go:29] interesting pod/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:49:16.750017 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:16.749991 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" podUID="cf9d00da-108f-4cd1-8ca8-5f89ba59c624" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:49:17.617223 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:17.617183 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:17.839453 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:17.839423 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:18.057045 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:18.056999 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:18.621551 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:18.621514 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:18.957039 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:18.956950 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:19.182312 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:19.182275 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:19.664153 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:19.649030 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:20.093291 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:20.093248 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:20.608393 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:20.608347 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:20.824565 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:20.824524 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:21.710874 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:21.710813 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:49:21.895721 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:21.895684 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istio-podinfo\") pod \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " Nov 02 21:49:21.896235 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:21.895742 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-kube-api-access-c9qtv\" (UniqueName: \"kubernetes.io/projected/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-kube-api-access-c9qtv\") pod \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " Nov 02 21:49:21.896235 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:21.895815 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-workload-certs\") pod \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " Nov 02 21:49:21.896235 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:21.895848 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-workload-socket\") pod \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " Nov 02 21:49:21.896235 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:21.895882 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istio-data\") pod \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " Nov 02 21:49:21.896235 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:21.895908 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istiod-ca-cert\") pod \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " Nov 02 21:49:21.896235 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:21.895988 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-credential-socket\") pod \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " Nov 02 21:49:21.896235 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:21.896021 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istio-token\" (UniqueName: \"kubernetes.io/projected/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istio-token\") pod \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " Nov 02 21:49:21.896235 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:21.896212 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istio-envoy\") pod \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\" (UID: \"cf9d00da-108f-4cd1-8ca8-5f89ba59c624\") " Nov 02 21:49:21.896811 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:21.896477 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-credential-socket" pod "cf9d00da-108f-4cd1-8ca8-5f89ba59c624" (UID: "cf9d00da-108f-4cd1-8ca8-5f89ba59c624"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:21.896896 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:21.896791 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istio-data" pod "cf9d00da-108f-4cd1-8ca8-5f89ba59c624" (UID: "cf9d00da-108f-4cd1-8ca8-5f89ba59c624"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:21.896896 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:21.896815 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:21.897111 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:21.897065 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-workload-socket" pod "cf9d00da-108f-4cd1-8ca8-5f89ba59c624" (UID: "cf9d00da-108f-4cd1-8ca8-5f89ba59c624"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:21.897275 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:21.897252 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istiod-ca-cert" pod "cf9d00da-108f-4cd1-8ca8-5f89ba59c624" (UID: "cf9d00da-108f-4cd1-8ca8-5f89ba59c624"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:49:21.897347 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:21.897316 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-workload-certs" pod "cf9d00da-108f-4cd1-8ca8-5f89ba59c624" (UID: "cf9d00da-108f-4cd1-8ca8-5f89ba59c624"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:21.898663 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:21.898631 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istio-podinfo" pod "cf9d00da-108f-4cd1-8ca8-5f89ba59c624" (UID: "cf9d00da-108f-4cd1-8ca8-5f89ba59c624"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:49:21.899029 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:21.899003 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istio-envoy" pod "cf9d00da-108f-4cd1-8ca8-5f89ba59c624" (UID: "cf9d00da-108f-4cd1-8ca8-5f89ba59c624"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:21.899207 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:21.899185 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-kube-api-access-c9qtv" pod "cf9d00da-108f-4cd1-8ca8-5f89ba59c624" (UID: "cf9d00da-108f-4cd1-8ca8-5f89ba59c624"). InnerVolumeSpecName "kube-api-access-c9qtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:21.899756 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:21.899735 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istio-token" pod "cf9d00da-108f-4cd1-8ca8-5f89ba59c624" (UID: "cf9d00da-108f-4cd1-8ca8-5f89ba59c624"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:21.997902 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:21.997825 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:21.997902 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:21.997853 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9qtv\" (UniqueName: \"kubernetes.io/projected/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-kube-api-access-c9qtv\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:21.997902 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:21.997863 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:21.997902 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:21.997872 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:21.997902 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:21.997881 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:21.997902 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:21.997890 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:21.997902 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:21.997897 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:21.997902 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:21.997905 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9d00da-108f-4cd1-8ca8-5f89ba59c624-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:22.378373 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:22.378336 4935 generic.go:356] "Generic (PLEG): container finished" podID="cf9d00da-108f-4cd1-8ca8-5f89ba59c624" containerID="1aaf70280cdebab112ecdae1e00e0f04bed5b20576660a87a38b527d0e281c30" exitCode=0 Nov 02 21:49:22.378575 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:22.378386 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" Nov 02 21:49:22.378575 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:22.378421 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" event={"ID":"cf9d00da-108f-4cd1-8ca8-5f89ba59c624","Type":"ContainerDied","Data":"1aaf70280cdebab112ecdae1e00e0f04bed5b20576660a87a38b527d0e281c30"} Nov 02 21:49:22.378575 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:22.378489 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4" event={"ID":"cf9d00da-108f-4cd1-8ca8-5f89ba59c624","Type":"ContainerDied","Data":"d62dc3a0d7b42b6a7729f15c116de4bc544b2ef3c7ff5e6a91462d5bf037a3bc"} Nov 02 21:49:22.378575 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:22.378514 4935 scope.go:122] "RemoveContainer" containerID="1aaf70280cdebab112ecdae1e00e0f04bed5b20576660a87a38b527d0e281c30" Nov 02 21:49:22.391264 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:22.391244 4935 scope.go:122] "RemoveContainer" containerID="e2f2517e6c5b39eed75d461e5d6f4bf0ee4fa50480678707a15766b101019baf" Nov 02 21:49:22.400530 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:22.400509 4935 scope.go:122] "RemoveContainer" containerID="1aaf70280cdebab112ecdae1e00e0f04bed5b20576660a87a38b527d0e281c30" Nov 02 21:49:22.400795 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:49:22.400776 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aaf70280cdebab112ecdae1e00e0f04bed5b20576660a87a38b527d0e281c30\": container with ID starting with 1aaf70280cdebab112ecdae1e00e0f04bed5b20576660a87a38b527d0e281c30 not found: ID does not exist" containerID="1aaf70280cdebab112ecdae1e00e0f04bed5b20576660a87a38b527d0e281c30" Nov 02 21:49:22.400853 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:22.400805 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aaf70280cdebab112ecdae1e00e0f04bed5b20576660a87a38b527d0e281c30"} err="failed to get container status \"1aaf70280cdebab112ecdae1e00e0f04bed5b20576660a87a38b527d0e281c30\": rpc error: code = NotFound desc = could not find container \"1aaf70280cdebab112ecdae1e00e0f04bed5b20576660a87a38b527d0e281c30\": container with ID starting with 1aaf70280cdebab112ecdae1e00e0f04bed5b20576660a87a38b527d0e281c30 not found: ID does not exist" Nov 02 21:49:22.400853 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:22.400825 4935 scope.go:122] "RemoveContainer" containerID="e2f2517e6c5b39eed75d461e5d6f4bf0ee4fa50480678707a15766b101019baf" Nov 02 21:49:22.401070 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:49:22.401049 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2f2517e6c5b39eed75d461e5d6f4bf0ee4fa50480678707a15766b101019baf\": container with ID starting with e2f2517e6c5b39eed75d461e5d6f4bf0ee4fa50480678707a15766b101019baf not found: ID does not exist" containerID="e2f2517e6c5b39eed75d461e5d6f4bf0ee4fa50480678707a15766b101019baf" Nov 02 21:49:22.401111 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:22.401080 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2f2517e6c5b39eed75d461e5d6f4bf0ee4fa50480678707a15766b101019baf"} err="failed to get container status \"e2f2517e6c5b39eed75d461e5d6f4bf0ee4fa50480678707a15766b101019baf\": rpc error: code = NotFound desc = could not find container \"e2f2517e6c5b39eed75d461e5d6f4bf0ee4fa50480678707a15766b101019baf\": container with ID starting with e2f2517e6c5b39eed75d461e5d6f4bf0ee4fa50480678707a15766b101019baf not found: ID does not exist" Nov 02 21:49:22.587266 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:22.587230 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4"] Nov 02 21:49:22.633685 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:22.633602 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-aa60bc-openshift-default-84f484f56c-pfnj4"] Nov 02 21:49:23.249672 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:23.249632 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:23.776349 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:23.776315 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf9d00da-108f-4cd1-8ca8-5f89ba59c624" path="/var/lib/kubelet/pods/cf9d00da-108f-4cd1-8ca8-5f89ba59c624/volumes" Nov 02 21:49:25.625300 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:25.625264 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:28.471214 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:49:28.471170 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:49:28.662016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:28.661965 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:28.899255 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:28.899212 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:29.143438 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:29.143402 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:29.575585 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:29.575549 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:29.794553 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:29.794518 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:31.152121 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:31.152086 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:31.379217 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:31.379183 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:32.398224 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:32.398189 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:32.622273 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:32.622233 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:34.492845 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:34.492814 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:38.380696 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:38.380620 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:38.603255 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:38.603173 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:40.085669 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:40.085635 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:45.354656 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:45.354586 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:50.733246 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:50.733208 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:55.742557 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:49:55.742506 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:49:58.469424 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:49:58.469381 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:50:00.768053 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:00.768006 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:01.089167 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:01.089063 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h"] Nov 02 21:50:01.089336 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:01.089311 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" podUID="52bdab02-9d8b-4a0f-90b2-bc9e705845f0" containerName="istio-proxy" containerID="cri-o://9ebbcd2dfa896bddf79e85b1719e7da5610fae83828dad7c0e2dbf5a39702362" gracePeriod=30 Nov 02 21:50:01.135694 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:01.135659 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r"] Nov 02 21:50:01.135908 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:01.135883 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" podUID="a22263a6-d470-4d3f-962c-955cd7b003c9" containerName="istio-proxy" containerID="cri-o://534a3f95412a8aa996372205742881b9c0ce212c31ac52b938d40b88224fe39d" gracePeriod=30 Nov 02 21:50:05.829364 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:05.829308 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:06.409069 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.409037 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:50:06.444395 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.444373 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:50:06.514002 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.513966 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-workload-certs\") pod \"a22263a6-d470-4d3f-962c-955cd7b003c9\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " Nov 02 21:50:06.514270 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.514066 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istio-data\") pod \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " Nov 02 21:50:06.514270 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.514111 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istio-envoy\") pod \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " Nov 02 21:50:06.514270 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.514183 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-istio-data\") pod \"a22263a6-d470-4d3f-962c-955cd7b003c9\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " Nov 02 21:50:06.514270 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.514223 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-credential-socket\") pod \"a22263a6-d470-4d3f-962c-955cd7b003c9\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " Nov 02 21:50:06.514270 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.514239 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-workload-certs" pod "a22263a6-d470-4d3f-962c-955cd7b003c9" (UID: "a22263a6-d470-4d3f-962c-955cd7b003c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:50:06.514567 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.514276 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-workload-certs\") pod \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " Nov 02 21:50:06.514567 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.514337 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istiod-ca-cert\") pod \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " Nov 02 21:50:06.514567 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.514371 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istio-data" pod "52bdab02-9d8b-4a0f-90b2-bc9e705845f0" (UID: "52bdab02-9d8b-4a0f-90b2-bc9e705845f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:50:06.514567 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.514381 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-kube-api-access-srdww\" (UniqueName: \"kubernetes.io/projected/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-kube-api-access-srdww\") pod \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " Nov 02 21:50:06.514567 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.514442 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istio-token\" (UniqueName: \"kubernetes.io/projected/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istio-token\") pod \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " Nov 02 21:50:06.514567 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.514485 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-istio-data" pod "a22263a6-d470-4d3f-962c-955cd7b003c9" (UID: "a22263a6-d470-4d3f-962c-955cd7b003c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:50:06.514567 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.514504 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-credential-socket" pod "a22263a6-d470-4d3f-962c-955cd7b003c9" (UID: "a22263a6-d470-4d3f-962c-955cd7b003c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:50:06.514567 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.514527 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a22263a6-d470-4d3f-962c-955cd7b003c9-istio-token\" (UniqueName: \"kubernetes.io/projected/a22263a6-d470-4d3f-962c-955cd7b003c9-istio-token\") pod \"a22263a6-d470-4d3f-962c-955cd7b003c9\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " Nov 02 21:50:06.514979 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.514571 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-credential-socket\") pod \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " Nov 02 21:50:06.514979 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.514625 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istio-podinfo\") pod \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " Nov 02 21:50:06.514979 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.514668 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a22263a6-d470-4d3f-962c-955cd7b003c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22263a6-d470-4d3f-962c-955cd7b003c9-istiod-ca-cert\") pod \"a22263a6-d470-4d3f-962c-955cd7b003c9\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " Nov 02 21:50:06.514979 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.514698 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-istio-envoy\") pod \"a22263a6-d470-4d3f-962c-955cd7b003c9\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " Nov 02 21:50:06.514979 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.514746 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-workload-socket\") pod \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\" (UID: \"52bdab02-9d8b-4a0f-90b2-bc9e705845f0\") " Nov 02 21:50:06.514979 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.514782 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a22263a6-d470-4d3f-962c-955cd7b003c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22263a6-d470-4d3f-962c-955cd7b003c9-istio-podinfo\") pod \"a22263a6-d470-4d3f-962c-955cd7b003c9\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " Nov 02 21:50:06.514979 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.514845 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a22263a6-d470-4d3f-962c-955cd7b003c9-kube-api-access-bfc5s\" (UniqueName: \"kubernetes.io/projected/a22263a6-d470-4d3f-962c-955cd7b003c9-kube-api-access-bfc5s\") pod \"a22263a6-d470-4d3f-962c-955cd7b003c9\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " Nov 02 21:50:06.514979 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.514871 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istiod-ca-cert" pod "52bdab02-9d8b-4a0f-90b2-bc9e705845f0" (UID: "52bdab02-9d8b-4a0f-90b2-bc9e705845f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:06.514979 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.514886 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-workload-socket\") pod \"a22263a6-d470-4d3f-962c-955cd7b003c9\" (UID: \"a22263a6-d470-4d3f-962c-955cd7b003c9\") " Nov 02 21:50:06.515462 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.515038 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-workload-socket" pod "a22263a6-d470-4d3f-962c-955cd7b003c9" (UID: "a22263a6-d470-4d3f-962c-955cd7b003c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:50:06.515462 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.515051 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-workload-certs" pod "52bdab02-9d8b-4a0f-90b2-bc9e705845f0" (UID: "52bdab02-9d8b-4a0f-90b2-bc9e705845f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:50:06.515462 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.515255 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-credential-socket" pod "52bdab02-9d8b-4a0f-90b2-bc9e705845f0" (UID: "52bdab02-9d8b-4a0f-90b2-bc9e705845f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:50:06.515462 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.515315 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a22263a6-d470-4d3f-962c-955cd7b003c9-istiod-ca-cert" pod "a22263a6-d470-4d3f-962c-955cd7b003c9" (UID: "a22263a6-d470-4d3f-962c-955cd7b003c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:06.515462 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.515441 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-workload-socket" pod "52bdab02-9d8b-4a0f-90b2-bc9e705845f0" (UID: "52bdab02-9d8b-4a0f-90b2-bc9e705845f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:50:06.515739 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.515580 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22263a6-d470-4d3f-962c-955cd7b003c9-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:06.515739 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.515605 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:06.515739 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.515626 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:06.515739 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.515641 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:06.515739 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.515655 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:06.515739 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.515668 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:06.515739 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.515686 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:06.515739 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.515707 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:06.515739 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.515738 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:06.516219 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.515754 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:06.517711 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.517673 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a22263a6-d470-4d3f-962c-955cd7b003c9-istio-podinfo" pod "a22263a6-d470-4d3f-962c-955cd7b003c9" (UID: "a22263a6-d470-4d3f-962c-955cd7b003c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:50:06.517837 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.517799 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istio-envoy" pod "52bdab02-9d8b-4a0f-90b2-bc9e705845f0" (UID: "52bdab02-9d8b-4a0f-90b2-bc9e705845f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:50:06.517837 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.517812 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istio-token" pod "52bdab02-9d8b-4a0f-90b2-bc9e705845f0" (UID: "52bdab02-9d8b-4a0f-90b2-bc9e705845f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:06.518093 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.518069 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-istio-envoy" pod "a22263a6-d470-4d3f-962c-955cd7b003c9" (UID: "a22263a6-d470-4d3f-962c-955cd7b003c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:50:06.518093 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.518078 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istio-podinfo" pod "52bdab02-9d8b-4a0f-90b2-bc9e705845f0" (UID: "52bdab02-9d8b-4a0f-90b2-bc9e705845f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:50:06.518255 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.518153 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-kube-api-access-srdww" pod "52bdab02-9d8b-4a0f-90b2-bc9e705845f0" (UID: "52bdab02-9d8b-4a0f-90b2-bc9e705845f0"). InnerVolumeSpecName "kube-api-access-srdww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:06.518989 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.518972 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22263a6-d470-4d3f-962c-955cd7b003c9-istio-token" pod "a22263a6-d470-4d3f-962c-955cd7b003c9" (UID: "a22263a6-d470-4d3f-962c-955cd7b003c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:06.519295 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.519279 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22263a6-d470-4d3f-962c-955cd7b003c9-kube-api-access-bfc5s" pod "a22263a6-d470-4d3f-962c-955cd7b003c9" (UID: "a22263a6-d470-4d3f-962c-955cd7b003c9"). InnerVolumeSpecName "kube-api-access-bfc5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:06.617441 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.617354 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:06.617441 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.617393 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a22263a6-d470-4d3f-962c-955cd7b003c9-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:06.617441 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.617408 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:06.617441 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.617425 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22263a6-d470-4d3f-962c-955cd7b003c9-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:06.617441 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.617439 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22263a6-d470-4d3f-962c-955cd7b003c9-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:06.617802 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.617455 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfc5s\" (UniqueName: \"kubernetes.io/projected/a22263a6-d470-4d3f-962c-955cd7b003c9-kube-api-access-bfc5s\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:06.617802 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.617469 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:06.617802 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.617483 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srdww\" (UniqueName: \"kubernetes.io/projected/52bdab02-9d8b-4a0f-90b2-bc9e705845f0-kube-api-access-srdww\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:06.658868 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.658831 4935 generic.go:356] "Generic (PLEG): container finished" podID="52bdab02-9d8b-4a0f-90b2-bc9e705845f0" containerID="9ebbcd2dfa896bddf79e85b1719e7da5610fae83828dad7c0e2dbf5a39702362" exitCode=0 Nov 02 21:50:06.659058 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.658881 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" Nov 02 21:50:06.659058 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.658922 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" event={"ID":"52bdab02-9d8b-4a0f-90b2-bc9e705845f0","Type":"ContainerDied","Data":"9ebbcd2dfa896bddf79e85b1719e7da5610fae83828dad7c0e2dbf5a39702362"} Nov 02 21:50:06.659058 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.658984 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h" event={"ID":"52bdab02-9d8b-4a0f-90b2-bc9e705845f0","Type":"ContainerDied","Data":"7c63c8074e004853198d0194932da0464e93aa3d39ef64e12b0e415231b87d88"} Nov 02 21:50:06.659058 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.659016 4935 scope.go:122] "RemoveContainer" containerID="9ebbcd2dfa896bddf79e85b1719e7da5610fae83828dad7c0e2dbf5a39702362" Nov 02 21:50:06.660748 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.660596 4935 generic.go:356] "Generic (PLEG): container finished" podID="a22263a6-d470-4d3f-962c-955cd7b003c9" containerID="534a3f95412a8aa996372205742881b9c0ce212c31ac52b938d40b88224fe39d" exitCode=0 Nov 02 21:50:06.660748 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.660646 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" Nov 02 21:50:06.660748 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.660646 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" event={"ID":"a22263a6-d470-4d3f-962c-955cd7b003c9","Type":"ContainerDied","Data":"534a3f95412a8aa996372205742881b9c0ce212c31ac52b938d40b88224fe39d"} Nov 02 21:50:06.660748 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.660696 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r" event={"ID":"a22263a6-d470-4d3f-962c-955cd7b003c9","Type":"ContainerDied","Data":"c5fd95a6edbdf59cf6cfdbcc5c4f5d8e4fc87a59f579d53d12edf3276b388363"} Nov 02 21:50:06.673782 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.673334 4935 scope.go:122] "RemoveContainer" containerID="9ebbcd2dfa896bddf79e85b1719e7da5610fae83828dad7c0e2dbf5a39702362" Nov 02 21:50:06.673782 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:50:06.673732 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ebbcd2dfa896bddf79e85b1719e7da5610fae83828dad7c0e2dbf5a39702362\": container with ID starting with 9ebbcd2dfa896bddf79e85b1719e7da5610fae83828dad7c0e2dbf5a39702362 not found: ID does not exist" containerID="9ebbcd2dfa896bddf79e85b1719e7da5610fae83828dad7c0e2dbf5a39702362" Nov 02 21:50:06.673782 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.673762 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ebbcd2dfa896bddf79e85b1719e7da5610fae83828dad7c0e2dbf5a39702362"} err="failed to get container status \"9ebbcd2dfa896bddf79e85b1719e7da5610fae83828dad7c0e2dbf5a39702362\": rpc error: code = NotFound desc = could not find container \"9ebbcd2dfa896bddf79e85b1719e7da5610fae83828dad7c0e2dbf5a39702362\": container with ID starting with 9ebbcd2dfa896bddf79e85b1719e7da5610fae83828dad7c0e2dbf5a39702362 not found: ID does not exist" Nov 02 21:50:06.673782 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.673783 4935 scope.go:122] "RemoveContainer" containerID="534a3f95412a8aa996372205742881b9c0ce212c31ac52b938d40b88224fe39d" Nov 02 21:50:06.688463 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.688439 4935 scope.go:122] "RemoveContainer" containerID="534a3f95412a8aa996372205742881b9c0ce212c31ac52b938d40b88224fe39d" Nov 02 21:50:06.688843 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:50:06.688815 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"534a3f95412a8aa996372205742881b9c0ce212c31ac52b938d40b88224fe39d\": container with ID starting with 534a3f95412a8aa996372205742881b9c0ce212c31ac52b938d40b88224fe39d not found: ID does not exist" containerID="534a3f95412a8aa996372205742881b9c0ce212c31ac52b938d40b88224fe39d" Nov 02 21:50:06.688955 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.688853 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"534a3f95412a8aa996372205742881b9c0ce212c31ac52b938d40b88224fe39d"} err="failed to get container status \"534a3f95412a8aa996372205742881b9c0ce212c31ac52b938d40b88224fe39d\": rpc error: code = NotFound desc = could not find container \"534a3f95412a8aa996372205742881b9c0ce212c31ac52b938d40b88224fe39d\": container with ID starting with 534a3f95412a8aa996372205742881b9c0ce212c31ac52b938d40b88224fe39d not found: ID does not exist" Nov 02 21:50:06.876498 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.876423 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h"] Nov 02 21:50:06.879907 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.879877 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-jvw8h"] Nov 02 21:50:06.983236 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.983202 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r"] Nov 02 21:50:06.987198 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:06.987170 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-t7k7r"] Nov 02 21:50:07.776562 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:07.776525 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52bdab02-9d8b-4a0f-90b2-bc9e705845f0" path="/var/lib/kubelet/pods/52bdab02-9d8b-4a0f-90b2-bc9e705845f0/volumes" Nov 02 21:50:07.777006 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:07.776993 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a22263a6-d470-4d3f-962c-955cd7b003c9" path="/var/lib/kubelet/pods/a22263a6-d470-4d3f-962c-955cd7b003c9/volumes" Nov 02 21:50:10.848082 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:10.848046 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:11.554897 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:11.554860 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:14.512513 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:14.512471 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:15.409668 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:15.409631 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr"] Nov 02 21:50:15.409883 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:15.409854 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr" podUID="d82a4ce3-ed44-4aa7-ad58-4c4a0233234b" containerName="maas-api" containerID="cri-o://099a72541ac8765554802311279fd53b75d1e256521c974b29c9c492988021d8" gracePeriod=30 Nov 02 21:50:15.608082 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:15.608047 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:15.714759 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:15.714735 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr" Nov 02 21:50:15.724851 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:15.724822 4935 generic.go:356] "Generic (PLEG): container finished" podID="d82a4ce3-ed44-4aa7-ad58-4c4a0233234b" containerID="099a72541ac8765554802311279fd53b75d1e256521c974b29c9c492988021d8" exitCode=0 Nov 02 21:50:15.724995 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:15.724875 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr" Nov 02 21:50:15.724995 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:15.724902 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr" event={"ID":"d82a4ce3-ed44-4aa7-ad58-4c4a0233234b","Type":"ContainerDied","Data":"099a72541ac8765554802311279fd53b75d1e256521c974b29c9c492988021d8"} Nov 02 21:50:15.724995 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:15.724951 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr" event={"ID":"d82a4ce3-ed44-4aa7-ad58-4c4a0233234b","Type":"ContainerDied","Data":"d92e5295b353294bf7356333bcca16ddcf06f53debabc1248b67b6b75169a287"} Nov 02 21:50:15.724995 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:15.724974 4935 scope.go:122] "RemoveContainer" containerID="099a72541ac8765554802311279fd53b75d1e256521c974b29c9c492988021d8" Nov 02 21:50:15.734509 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:15.734483 4935 scope.go:122] "RemoveContainer" containerID="099a72541ac8765554802311279fd53b75d1e256521c974b29c9c492988021d8" Nov 02 21:50:15.734753 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:50:15.734730 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"099a72541ac8765554802311279fd53b75d1e256521c974b29c9c492988021d8\": container with ID starting with 099a72541ac8765554802311279fd53b75d1e256521c974b29c9c492988021d8 not found: ID does not exist" containerID="099a72541ac8765554802311279fd53b75d1e256521c974b29c9c492988021d8" Nov 02 21:50:15.734837 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:15.734764 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"099a72541ac8765554802311279fd53b75d1e256521c974b29c9c492988021d8"} err="failed to get container status \"099a72541ac8765554802311279fd53b75d1e256521c974b29c9c492988021d8\": rpc error: code = NotFound desc = could not find container \"099a72541ac8765554802311279fd53b75d1e256521c974b29c9c492988021d8\": container with ID starting with 099a72541ac8765554802311279fd53b75d1e256521c974b29c9c492988021d8 not found: ID does not exist" Nov 02 21:50:15.814865 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:15.814833 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-maas-api-tls\") pod \"d82a4ce3-ed44-4aa7-ad58-4c4a0233234b\" (UID: \"d82a4ce3-ed44-4aa7-ad58-4c4a0233234b\") " Nov 02 21:50:15.815073 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:15.814922 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-kube-api-access-7z7zj\" (UniqueName: \"kubernetes.io/projected/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-kube-api-access-7z7zj\") pod \"d82a4ce3-ed44-4aa7-ad58-4c4a0233234b\" (UID: \"d82a4ce3-ed44-4aa7-ad58-4c4a0233234b\") " Nov 02 21:50:15.815163 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:15.815087 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-openshift-service-ca\") pod \"d82a4ce3-ed44-4aa7-ad58-4c4a0233234b\" (UID: \"d82a4ce3-ed44-4aa7-ad58-4c4a0233234b\") " Nov 02 21:50:15.815163 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:15.815122 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-metrics-certs\") pod \"d82a4ce3-ed44-4aa7-ad58-4c4a0233234b\" (UID: \"d82a4ce3-ed44-4aa7-ad58-4c4a0233234b\") " Nov 02 21:50:15.815664 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:15.815620 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-openshift-service-ca" pod "d82a4ce3-ed44-4aa7-ad58-4c4a0233234b" (UID: "d82a4ce3-ed44-4aa7-ad58-4c4a0233234b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:15.827164 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:15.820946 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-metrics-certs" pod "d82a4ce3-ed44-4aa7-ad58-4c4a0233234b" (UID: "d82a4ce3-ed44-4aa7-ad58-4c4a0233234b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:15.827538 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:15.827481 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-kube-api-access-7z7zj" pod "d82a4ce3-ed44-4aa7-ad58-4c4a0233234b" (UID: "d82a4ce3-ed44-4aa7-ad58-4c4a0233234b"). InnerVolumeSpecName "kube-api-access-7z7zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:15.843347 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:15.843305 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-maas-api-tls" pod "d82a4ce3-ed44-4aa7-ad58-4c4a0233234b" (UID: "d82a4ce3-ed44-4aa7-ad58-4c4a0233234b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:15.916961 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:15.916929 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:15.916961 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:15.916962 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z7zj\" (UniqueName: \"kubernetes.io/projected/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-kube-api-access-7z7zj\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:15.917182 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:15.916977 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:15.917182 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:15.916991 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:16.139219 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:16.139185 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr"] Nov 02 21:50:16.141604 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:16.141583 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cc0d6a-6f96866fb4-r7cnr"] Nov 02 21:50:17.427736 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:17.427698 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:17.777736 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:17.777695 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d82a4ce3-ed44-4aa7-ad58-4c4a0233234b" path="/var/lib/kubelet/pods/d82a4ce3-ed44-4aa7-ad58-4c4a0233234b/volumes" Nov 02 21:50:20.328630 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:20.328573 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:20.546170 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:20.541314 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:20.748044 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:20.747948 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:22.632669 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:22.632638 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:23.377478 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:23.377436 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:23.603878 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:23.603842 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:24.901078 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:24.901034 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:25.393900 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:25.393865 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:26.342563 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:26.342516 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:28.075769 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:28.075730 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:28.292829 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:28.292793 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:28.471169 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:50:28.471061 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:50:28.494622 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:28.494592 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:31.477481 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:31.477449 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:32.416273 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:32.416223 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:33.769532 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:33.769493 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:37.612269 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:37.612230 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:38.687277 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:38.687238 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:38.709571 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:38.709543 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn"] Nov 02 21:50:38.709773 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:38.709753 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" podUID="430fb50a-4cbe-4b91-9e00-7484df623aec" containerName="istio-proxy" containerID="cri-o://1888087af1eb35a768641da3e93ecbf331fbbfc1f2d02ea8868bc75611aff1ca" gracePeriod=30 Nov 02 21:50:38.951081 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:38.950999 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:39.046723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:39.046688 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2gjqn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:50:39.046895 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:39.046752 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" podUID="430fb50a-4cbe-4b91-9e00-7484df623aec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:50:43.916750 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:43.916716 4935 generic.go:356] "Generic (PLEG): container finished" podID="430fb50a-4cbe-4b91-9e00-7484df623aec" containerID="1888087af1eb35a768641da3e93ecbf331fbbfc1f2d02ea8868bc75611aff1ca" exitCode=0 Nov 02 21:50:43.917285 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:43.916938 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" event={"ID":"430fb50a-4cbe-4b91-9e00-7484df623aec","Type":"ContainerDied","Data":"1888087af1eb35a768641da3e93ecbf331fbbfc1f2d02ea8868bc75611aff1ca"} Nov 02 21:50:43.949610 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:43.949576 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:43.987807 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:43.987738 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:50:44.137669 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.137630 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/430fb50a-4cbe-4b91-9e00-7484df623aec-kube-api-access-v56zq\" (UniqueName: \"kubernetes.io/projected/430fb50a-4cbe-4b91-9e00-7484df623aec-kube-api-access-v56zq\") pod \"430fb50a-4cbe-4b91-9e00-7484df623aec\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " Nov 02 21:50:44.137869 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.137680 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/430fb50a-4cbe-4b91-9e00-7484df623aec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/430fb50a-4cbe-4b91-9e00-7484df623aec-istio-podinfo\") pod \"430fb50a-4cbe-4b91-9e00-7484df623aec\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " Nov 02 21:50:44.137869 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.137774 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-istio-envoy\") pod \"430fb50a-4cbe-4b91-9e00-7484df623aec\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " Nov 02 21:50:44.137869 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.137854 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-workload-certs\") pod \"430fb50a-4cbe-4b91-9e00-7484df623aec\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " Nov 02 21:50:44.138048 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.137885 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-workload-socket\") pod \"430fb50a-4cbe-4b91-9e00-7484df623aec\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " Nov 02 21:50:44.138048 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.137918 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-istio-data\") pod \"430fb50a-4cbe-4b91-9e00-7484df623aec\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " Nov 02 21:50:44.138048 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.137970 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/430fb50a-4cbe-4b91-9e00-7484df623aec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/430fb50a-4cbe-4b91-9e00-7484df623aec-istiod-ca-cert\") pod \"430fb50a-4cbe-4b91-9e00-7484df623aec\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " Nov 02 21:50:44.138048 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.138020 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/430fb50a-4cbe-4b91-9e00-7484df623aec-istio-token\" (UniqueName: \"kubernetes.io/projected/430fb50a-4cbe-4b91-9e00-7484df623aec-istio-token\") pod \"430fb50a-4cbe-4b91-9e00-7484df623aec\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " Nov 02 21:50:44.138291 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.138068 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-credential-socket\") pod \"430fb50a-4cbe-4b91-9e00-7484df623aec\" (UID: \"430fb50a-4cbe-4b91-9e00-7484df623aec\") " Nov 02 21:50:44.138291 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.138125 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-workload-certs" pod "430fb50a-4cbe-4b91-9e00-7484df623aec" (UID: "430fb50a-4cbe-4b91-9e00-7484df623aec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:50:44.138291 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.138208 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-workload-socket" pod "430fb50a-4cbe-4b91-9e00-7484df623aec" (UID: "430fb50a-4cbe-4b91-9e00-7484df623aec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:50:44.138462 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.138440 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/430fb50a-4cbe-4b91-9e00-7484df623aec-istiod-ca-cert" pod "430fb50a-4cbe-4b91-9e00-7484df623aec" (UID: "430fb50a-4cbe-4b91-9e00-7484df623aec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:44.138542 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.138513 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-istio-data" pod "430fb50a-4cbe-4b91-9e00-7484df623aec" (UID: "430fb50a-4cbe-4b91-9e00-7484df623aec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:50:44.138542 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.138527 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-credential-socket" pod "430fb50a-4cbe-4b91-9e00-7484df623aec" (UID: "430fb50a-4cbe-4b91-9e00-7484df623aec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:50:44.138644 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.138606 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:44.138644 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.138626 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:44.138721 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.138641 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/430fb50a-4cbe-4b91-9e00-7484df623aec-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:44.140048 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.140023 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/430fb50a-4cbe-4b91-9e00-7484df623aec-kube-api-access-v56zq" pod "430fb50a-4cbe-4b91-9e00-7484df623aec" (UID: "430fb50a-4cbe-4b91-9e00-7484df623aec"). InnerVolumeSpecName "kube-api-access-v56zq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:44.140232 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.140054 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/430fb50a-4cbe-4b91-9e00-7484df623aec-istio-podinfo" pod "430fb50a-4cbe-4b91-9e00-7484df623aec" (UID: "430fb50a-4cbe-4b91-9e00-7484df623aec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:50:44.140232 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.140124 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/430fb50a-4cbe-4b91-9e00-7484df623aec-istio-token" pod "430fb50a-4cbe-4b91-9e00-7484df623aec" (UID: "430fb50a-4cbe-4b91-9e00-7484df623aec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:44.140523 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.140506 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-istio-envoy" pod "430fb50a-4cbe-4b91-9e00-7484df623aec" (UID: "430fb50a-4cbe-4b91-9e00-7484df623aec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:50:44.239454 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.239369 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:44.239454 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.239402 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/430fb50a-4cbe-4b91-9e00-7484df623aec-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:44.239454 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.239415 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:44.239454 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.239429 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v56zq\" (UniqueName: \"kubernetes.io/projected/430fb50a-4cbe-4b91-9e00-7484df623aec-kube-api-access-v56zq\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:44.239454 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.239442 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/430fb50a-4cbe-4b91-9e00-7484df623aec-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:44.239454 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.239454 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/430fb50a-4cbe-4b91-9e00-7484df623aec-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:44.924176 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.924122 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" event={"ID":"430fb50a-4cbe-4b91-9e00-7484df623aec","Type":"ContainerDied","Data":"8050284d67b8ae95dda03f5d8f9ab54013baee9b34b13b05190fa504c644f73c"} Nov 02 21:50:44.924640 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.924186 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn" Nov 02 21:50:44.924640 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:44.924210 4935 scope.go:122] "RemoveContainer" containerID="1888087af1eb35a768641da3e93ecbf331fbbfc1f2d02ea8868bc75611aff1ca" Nov 02 21:50:45.078147 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:45.078109 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn"] Nov 02 21:50:45.124269 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:45.124236 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2gjqn"] Nov 02 21:50:45.776865 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:45.776828 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="430fb50a-4cbe-4b91-9e00-7484df623aec" path="/var/lib/kubelet/pods/430fb50a-4cbe-4b91-9e00-7484df623aec/volumes" Nov 02 21:50:49.087434 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:49.087393 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:54.218456 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:54.218421 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:50:58.472978 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:50:58.472938 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:50:58.632771 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:50:58.632726 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:00.406527 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:00.406462 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:05.450003 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:05.449960 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:06.570449 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:06.570413 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:06.790835 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:06.790793 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:08.134177 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:08.134063 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:08.577743 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:08.577712 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:09.360207 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:09.360154 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:10.972202 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:10.972165 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:15.033577 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:15.033538 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:16.180489 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:16.180446 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:16.393027 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:16.392989 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:16.635246 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:16.635214 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:17.259632 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:17.259591 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:17.468030 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:17.467992 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:20.596951 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:20.596916 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:21.836239 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:21.836150 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:26.934849 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:26.934812 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:28.470560 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:51:28.470513 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:51:29.548507 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:29.548473 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:30.698924 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:30.698885 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:31.119378 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.119342 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q"] Nov 02 21:51:31.120071 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.120044 4935 cpu_manager.go:414] "containerMap: removing container" podUID="ff38001c-9b28-473a-bdc7-9ebc8514fe51" containerName="istio-proxy" Nov 02 21:51:31.120071 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.120069 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff38001c-9b28-473a-bdc7-9ebc8514fe51" containerName="istio-proxy" Nov 02 21:51:31.120071 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.120086 4935 cpu_manager.go:414] "containerMap: removing container" podUID="52bdab02-9d8b-4a0f-90b2-bc9e705845f0" containerName="istio-proxy" Nov 02 21:51:31.120071 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.120094 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52bdab02-9d8b-4a0f-90b2-bc9e705845f0" containerName="istio-proxy" Nov 02 21:51:31.120492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.120147 4935 cpu_manager.go:414] "containerMap: removing container" podUID="d82a4ce3-ed44-4aa7-ad58-4c4a0233234b" containerName="maas-api" Nov 02 21:51:31.120492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.120156 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d82a4ce3-ed44-4aa7-ad58-4c4a0233234b" containerName="maas-api" Nov 02 21:51:31.120492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.120173 4935 cpu_manager.go:414] "containerMap: removing container" podUID="430fb50a-4cbe-4b91-9e00-7484df623aec" containerName="istio-proxy" Nov 02 21:51:31.120492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.120183 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="430fb50a-4cbe-4b91-9e00-7484df623aec" containerName="istio-proxy" Nov 02 21:51:31.120492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.120201 4935 cpu_manager.go:414] "containerMap: removing container" podUID="a22263a6-d470-4d3f-962c-955cd7b003c9" containerName="istio-proxy" Nov 02 21:51:31.120492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.120208 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a22263a6-d470-4d3f-962c-955cd7b003c9" containerName="istio-proxy" Nov 02 21:51:31.120492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.120217 4935 cpu_manager.go:414] "containerMap: removing container" podUID="cf9d00da-108f-4cd1-8ca8-5f89ba59c624" containerName="istio-proxy" Nov 02 21:51:31.120492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.120224 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf9d00da-108f-4cd1-8ca8-5f89ba59c624" containerName="istio-proxy" Nov 02 21:51:31.120492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.120236 4935 cpu_manager.go:414] "containerMap: removing container" podUID="cf9d00da-108f-4cd1-8ca8-5f89ba59c624" containerName="istio-proxy" Nov 02 21:51:31.120492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.120243 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf9d00da-108f-4cd1-8ca8-5f89ba59c624" containerName="istio-proxy" Nov 02 21:51:31.120492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.120408 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="d82a4ce3-ed44-4aa7-ad58-4c4a0233234b" containerName="maas-api" Nov 02 21:51:31.120492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.120422 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="430fb50a-4cbe-4b91-9e00-7484df623aec" containerName="istio-proxy" Nov 02 21:51:31.120492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.120431 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="a22263a6-d470-4d3f-962c-955cd7b003c9" containerName="istio-proxy" Nov 02 21:51:31.120492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.120443 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf9d00da-108f-4cd1-8ca8-5f89ba59c624" containerName="istio-proxy" Nov 02 21:51:31.120492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.120453 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf9d00da-108f-4cd1-8ca8-5f89ba59c624" containerName="istio-proxy" Nov 02 21:51:31.120492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.120461 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff38001c-9b28-473a-bdc7-9ebc8514fe51" containerName="istio-proxy" Nov 02 21:51:31.120492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.120472 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff38001c-9b28-473a-bdc7-9ebc8514fe51" containerName="istio-proxy" Nov 02 21:51:31.120492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.120481 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="52bdab02-9d8b-4a0f-90b2-bc9e705845f0" containerName="istio-proxy" Nov 02 21:51:31.131454 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.131424 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.225828 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.210722 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q"] Nov 02 21:51:31.294599 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.294560 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.294818 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.294624 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83930246-18af-44a1-b4c6-3353bbba6814-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.294818 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.294688 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.294961 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.294933 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83930246-18af-44a1-b4c6-3353bbba6814-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.295101 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.295080 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.295343 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.295322 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kz7s\" (UniqueName: \"kubernetes.io/projected/83930246-18af-44a1-b4c6-3353bbba6814-kube-api-access-5kz7s\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.295644 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.295624 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.295908 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.295889 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83930246-18af-44a1-b4c6-3353bbba6814-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.296198 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.296166 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.307171 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.305371 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:31.398283 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.398179 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.398283 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.398249 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83930246-18af-44a1-b4c6-3353bbba6814-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.398519 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.398324 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.398519 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.398359 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83930246-18af-44a1-b4c6-3353bbba6814-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.398519 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.398401 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.398519 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.398430 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kz7s\" (UniqueName: \"kubernetes.io/projected/83930246-18af-44a1-b4c6-3353bbba6814-kube-api-access-5kz7s\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.398519 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.398459 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.398519 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.398500 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83930246-18af-44a1-b4c6-3353bbba6814-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.398832 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.398542 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.398832 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.398706 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.399081 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.399057 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.399538 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.399511 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.399628 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.399550 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.400173 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.400122 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83930246-18af-44a1-b4c6-3353bbba6814-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.400809 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.400787 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.401449 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.401424 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83930246-18af-44a1-b4c6-3353bbba6814-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.465978 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.465246 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83930246-18af-44a1-b4c6-3353bbba6814-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.466889 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.466863 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kz7s\" (UniqueName: \"kubernetes.io/projected/83930246-18af-44a1-b4c6-3353bbba6814-kube-api-access-5kz7s\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.619200 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.619167 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:31.757110 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.757023 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:31.817653 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:31.817616 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:32.182980 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:32.182942 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q"] Nov 02 21:51:32.186585 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:51:32.186553 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83930246_18af_44a1_b4c6_3353bbba6814.slice/crio-e77ab65bd584eb3015cdb53bc19842ff2acdd72fa1e49c7ec407f638445fba9d WatchSource:0}: Error finding container e77ab65bd584eb3015cdb53bc19842ff2acdd72fa1e49c7ec407f638445fba9d: Status 404 returned error can't find the container with id e77ab65bd584eb3015cdb53bc19842ff2acdd72fa1e49c7ec407f638445fba9d Nov 02 21:51:32.188980 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:32.188944 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:32.189077 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:32.189019 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:32.189077 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:32.189053 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:32.312567 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:32.312529 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" event={"ID":"83930246-18af-44a1-b4c6-3353bbba6814","Type":"ContainerStarted","Data":"2c2e7b8357b6413a93294126a40ad5f6d80736f8be5fa69c8deaffcd6a9298b6"} Nov 02 21:51:32.312753 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:32.312584 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" event={"ID":"83930246-18af-44a1-b4c6-3353bbba6814","Type":"ContainerStarted","Data":"e77ab65bd584eb3015cdb53bc19842ff2acdd72fa1e49c7ec407f638445fba9d"} Nov 02 21:51:32.315641 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:32.315603 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:32.424051 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:32.423993 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" podStartSLOduration=1.423977823 podStartE2EDuration="1.423977823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:51:32.420663086 +0000 UTC m=+2584.860237808" watchObservedRunningTime="2026-11-02 21:51:32.423977823 +0000 UTC m=+2584.863552544" Nov 02 21:51:32.757360 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:32.757313 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:33.760100 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:33.760062 4935 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 21:51:33.760598 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:33.760140 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" podUID="83930246-18af-44a1-b4c6-3353bbba6814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 21:51:34.757888 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:34.757849 4935 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 21:51:34.758080 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:34.757913 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" podUID="83930246-18af-44a1-b4c6-3353bbba6814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 21:51:35.555483 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:35.555448 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:35.758020 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:35.757981 4935 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 21:51:35.758227 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:35.758049 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" podUID="83930246-18af-44a1-b4c6-3353bbba6814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 21:51:35.787246 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:35.787206 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:36.757742 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:36.757703 4935 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 21:51:36.758175 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:36.757769 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" podUID="83930246-18af-44a1-b4c6-3353bbba6814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 21:51:37.757698 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:37.757662 4935 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 21:51:37.757904 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:37.757719 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" podUID="83930246-18af-44a1-b4c6-3353bbba6814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 21:51:38.022640 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:38.022594 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:38.746167 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:38.745265 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:38.772394 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:38.772353 4935 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 21:51:38.772889 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:38.772417 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" podUID="83930246-18af-44a1-b4c6-3353bbba6814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 21:51:38.976367 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:38.976260 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:39.762827 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:39.762784 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:39.764029 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:39.763997 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:39.764677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:39.764657 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 21:51:43.422594 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:43.422563 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:48.462292 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:48.462257 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:52.638397 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:52.638366 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v"] Nov 02 21:51:52.638878 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:52.638644 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v" podUID="8210d255-0b34-442c-b473-87051911a385" containerName="maas-api" containerID="cri-o://e86047cd896d589be579b658a2b11e47a940a823be824992ba2de806ba9ec9d9" gracePeriod=30 Nov 02 21:51:52.815976 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:52.815045 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:52.914887 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:52.914859 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v" Nov 02 21:51:53.073576 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:53.073535 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8210d255-0b34-442c-b473-87051911a385-kube-api-access-4zjzw\" (UniqueName: \"kubernetes.io/projected/8210d255-0b34-442c-b473-87051911a385-kube-api-access-4zjzw\") pod \"8210d255-0b34-442c-b473-87051911a385\" (UID: \"8210d255-0b34-442c-b473-87051911a385\") " Nov 02 21:51:53.073790 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:53.073687 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8210d255-0b34-442c-b473-87051911a385-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8210d255-0b34-442c-b473-87051911a385-maas-api-tls\") pod \"8210d255-0b34-442c-b473-87051911a385\" (UID: \"8210d255-0b34-442c-b473-87051911a385\") " Nov 02 21:51:53.073790 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:53.073761 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8210d255-0b34-442c-b473-87051911a385-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8210d255-0b34-442c-b473-87051911a385-openshift-service-ca\") pod \"8210d255-0b34-442c-b473-87051911a385\" (UID: \"8210d255-0b34-442c-b473-87051911a385\") " Nov 02 21:51:53.073915 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:53.073859 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8210d255-0b34-442c-b473-87051911a385-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8210d255-0b34-442c-b473-87051911a385-metrics-certs\") pod \"8210d255-0b34-442c-b473-87051911a385\" (UID: \"8210d255-0b34-442c-b473-87051911a385\") " Nov 02 21:51:53.074833 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:53.074800 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8210d255-0b34-442c-b473-87051911a385-openshift-service-ca" pod "8210d255-0b34-442c-b473-87051911a385" (UID: "8210d255-0b34-442c-b473-87051911a385"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:51:53.081042 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:53.079526 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8210d255-0b34-442c-b473-87051911a385-maas-api-tls" pod "8210d255-0b34-442c-b473-87051911a385" (UID: "8210d255-0b34-442c-b473-87051911a385"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:51:53.092326 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:53.081618 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8210d255-0b34-442c-b473-87051911a385-metrics-certs" pod "8210d255-0b34-442c-b473-87051911a385" (UID: "8210d255-0b34-442c-b473-87051911a385"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:51:53.092326 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:53.081793 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8210d255-0b34-442c-b473-87051911a385-kube-api-access-4zjzw" pod "8210d255-0b34-442c-b473-87051911a385" (UID: "8210d255-0b34-442c-b473-87051911a385"). InnerVolumeSpecName "kube-api-access-4zjzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:51:53.175171 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:53.175079 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zjzw\" (UniqueName: \"kubernetes.io/projected/8210d255-0b34-442c-b473-87051911a385-kube-api-access-4zjzw\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:51:53.175171 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:53.175111 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8210d255-0b34-442c-b473-87051911a385-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:51:53.175171 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:53.175120 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8210d255-0b34-442c-b473-87051911a385-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:51:53.175171 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:53.175143 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8210d255-0b34-442c-b473-87051911a385-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:51:53.482126 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:53.482046 4935 generic.go:356] "Generic (PLEG): container finished" podID="8210d255-0b34-442c-b473-87051911a385" containerID="e86047cd896d589be579b658a2b11e47a940a823be824992ba2de806ba9ec9d9" exitCode=0 Nov 02 21:51:53.482126 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:53.482091 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v" Nov 02 21:51:53.482126 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:53.482116 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v" event={"ID":"8210d255-0b34-442c-b473-87051911a385","Type":"ContainerDied","Data":"e86047cd896d589be579b658a2b11e47a940a823be824992ba2de806ba9ec9d9"} Nov 02 21:51:53.482421 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:53.482178 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v" event={"ID":"8210d255-0b34-442c-b473-87051911a385","Type":"ContainerDied","Data":"1e0248b54e8666c8d940311d791f0405f8d2bdca8f0a012745af4e8ade6658ba"} Nov 02 21:51:53.482421 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:53.482202 4935 scope.go:122] "RemoveContainer" containerID="e86047cd896d589be579b658a2b11e47a940a823be824992ba2de806ba9ec9d9" Nov 02 21:51:53.495546 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:53.495486 4935 scope.go:122] "RemoveContainer" containerID="e86047cd896d589be579b658a2b11e47a940a823be824992ba2de806ba9ec9d9" Nov 02 21:51:53.495942 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:51:53.495918 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e86047cd896d589be579b658a2b11e47a940a823be824992ba2de806ba9ec9d9\": container with ID starting with e86047cd896d589be579b658a2b11e47a940a823be824992ba2de806ba9ec9d9 not found: ID does not exist" containerID="e86047cd896d589be579b658a2b11e47a940a823be824992ba2de806ba9ec9d9" Nov 02 21:51:53.496024 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:53.495952 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e86047cd896d589be579b658a2b11e47a940a823be824992ba2de806ba9ec9d9"} err="failed to get container status \"e86047cd896d589be579b658a2b11e47a940a823be824992ba2de806ba9ec9d9\": rpc error: code = NotFound desc = could not find container \"e86047cd896d589be579b658a2b11e47a940a823be824992ba2de806ba9ec9d9\": container with ID starting with e86047cd896d589be579b658a2b11e47a940a823be824992ba2de806ba9ec9d9 not found: ID does not exist" Nov 02 21:51:53.637122 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:53.637085 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v"] Nov 02 21:51:53.682642 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:53.682597 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-942164-644bb98c7-v7q2v"] Nov 02 21:51:53.775908 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:53.775877 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8210d255-0b34-442c-b473-87051911a385" path="/var/lib/kubelet/pods/8210d255-0b34-442c-b473-87051911a385/volumes" Nov 02 21:51:54.626618 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:54.626584 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:51:58.468499 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:51:58.468459 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:51:59.632075 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:51:59.632034 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:00.840353 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:00.840319 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:01.127320 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.127235 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd"] Nov 02 21:52:01.127750 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.127737 4935 cpu_manager.go:414] "containerMap: removing container" podUID="8210d255-0b34-442c-b473-87051911a385" containerName="maas-api" Nov 02 21:52:01.127808 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.127751 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8210d255-0b34-442c-b473-87051911a385" containerName="maas-api" Nov 02 21:52:01.127808 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.127766 4935 cpu_manager.go:414] "containerMap: removing container" podUID="ff38001c-9b28-473a-bdc7-9ebc8514fe51" containerName="istio-proxy" Nov 02 21:52:01.127808 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.127772 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff38001c-9b28-473a-bdc7-9ebc8514fe51" containerName="istio-proxy" Nov 02 21:52:01.127904 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.127848 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="8210d255-0b34-442c-b473-87051911a385" containerName="maas-api" Nov 02 21:52:01.131362 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.131337 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.157674 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.157638 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.157851 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.157685 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.157851 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.157710 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.157851 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.157839 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.158033 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.157877 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.158033 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.157910 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vhwb\" (UniqueName: \"kubernetes.io/projected/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-kube-api-access-9vhwb\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.158033 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.157979 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.158235 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.158037 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.158235 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.158069 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.173420 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.173389 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p"] Nov 02 21:52:01.177311 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.177287 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.181665 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.181647 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-0c9014-openshift-default-dockercfg-7ll9g\"" Nov 02 21:52:01.195842 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.195816 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:01.225433 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.225405 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd"] Nov 02 21:52:01.232353 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.232316 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p"] Nov 02 21:52:01.258716 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.258681 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfmsh\" (UniqueName: \"kubernetes.io/projected/7caa0671-29e2-468b-9dc2-d125ecd7ba84-kube-api-access-bfmsh\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.258927 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.258729 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.258927 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.258759 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.258927 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.258851 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.258927 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.258898 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.258927 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.258923 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.259294 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.258986 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vhwb\" (UniqueName: \"kubernetes.io/projected/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-kube-api-access-9vhwb\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.259294 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.259149 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.259294 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.259205 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.259294 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.259236 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.259294 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.259274 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.259563 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.259358 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.259563 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.259391 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.259563 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.259431 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.259563 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.259473 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.259563 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.259505 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.259563 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.259523 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.259563 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.259553 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.259940 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.259759 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.259940 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.259923 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.260051 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.260006 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.260116 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.260060 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.260380 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.260357 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.262163 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.262108 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.262394 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.262365 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.312713 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.312668 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.312872 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.312676 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vhwb\" (UniqueName: \"kubernetes.io/projected/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-kube-api-access-9vhwb\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.360901 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.360863 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.361090 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.360950 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.361090 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.360977 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.361090 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.361020 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.361090 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.361048 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.361090 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.361084 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfmsh\" (UniqueName: \"kubernetes.io/projected/7caa0671-29e2-468b-9dc2-d125ecd7ba84-kube-api-access-bfmsh\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.361394 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.361236 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.361394 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.361274 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.361394 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.361301 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.361646 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.361616 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-workload-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.361737 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.361706 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-workload-certs\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.361929 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.361905 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-credential-socket\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.362032 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.361975 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istio-data\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.362188 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.362121 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istiod-ca-cert\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.364245 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.364220 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istio-envoy\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.364448 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.364426 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istio-podinfo\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.414113 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.414035 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfmsh\" (UniqueName: \"kubernetes.io/projected/7caa0671-29e2-468b-9dc2-d125ecd7ba84-kube-api-access-bfmsh\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.414113 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.414059 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istio-token\") pod \"e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.447497 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.447461 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:01.491047 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.491013 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:01.865086 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.865044 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd"] Nov 02 21:52:01.905278 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:52:01.905246 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd064c27a_7a1c_4fe2_ad81_9db92ce9d6ee.slice/crio-2ea36479975c811184dbfff0c2ce02dc3f2b33c8e9dcf4ce049e1bd37d4a2d01 WatchSource:0}: Error finding container 2ea36479975c811184dbfff0c2ce02dc3f2b33c8e9dcf4ce049e1bd37d4a2d01: Status 404 returned error can't find the container with id 2ea36479975c811184dbfff0c2ce02dc3f2b33c8e9dcf4ce049e1bd37d4a2d01 Nov 02 21:52:01.907656 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.907623 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:52:01.907732 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.907690 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:52:01.907732 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.907721 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:52:01.916275 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.916249 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p"] Nov 02 21:52:01.918584 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:52:01.918556 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7caa0671_29e2_468b_9dc2_d125ecd7ba84.slice/crio-97b9b1539dcd90f667d55b879a03bc1c71ea1cdba64cc7a63866a117ec31de52 WatchSource:0}: Error finding container 97b9b1539dcd90f667d55b879a03bc1c71ea1cdba64cc7a63866a117ec31de52: Status 404 returned error can't find the container with id 97b9b1539dcd90f667d55b879a03bc1c71ea1cdba64cc7a63866a117ec31de52 Nov 02 21:52:01.921090 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.921060 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:52:01.921206 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.921122 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:52:01.921251 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:01.921230 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:52:02.469457 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:02.469417 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:02.562153 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:02.561975 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" event={"ID":"7caa0671-29e2-468b-9dc2-d125ecd7ba84","Type":"ContainerStarted","Data":"0a957eed5d2c2e83955d9e66dcb33663ba52edaa6c76e1119a2d5465c34ec8cf"} Nov 02 21:52:02.562153 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:02.562041 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" event={"ID":"7caa0671-29e2-468b-9dc2-d125ecd7ba84","Type":"ContainerStarted","Data":"97b9b1539dcd90f667d55b879a03bc1c71ea1cdba64cc7a63866a117ec31de52"} Nov 02 21:52:02.581638 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:02.570019 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" event={"ID":"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee","Type":"ContainerStarted","Data":"5aca8b81fcf8be259a0920c0f16042145ea27df2578234539a72ccc90f13b1b9"} Nov 02 21:52:02.581638 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:02.570087 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" event={"ID":"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee","Type":"ContainerStarted","Data":"2ea36479975c811184dbfff0c2ce02dc3f2b33c8e9dcf4ce049e1bd37d4a2d01"} Nov 02 21:52:02.677155 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:02.676866 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" podStartSLOduration=1.676848123 podStartE2EDuration="1.676848123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:52:02.672963211 +0000 UTC m=+2615.112537936" watchObservedRunningTime="2026-11-02 21:52:02.676848123 +0000 UTC m=+2615.116422840" Nov 02 21:52:02.785486 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:02.785408 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" podStartSLOduration=1.785388183 podStartE2EDuration="1.785388183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:52:02.782717028 +0000 UTC m=+2615.222291752" watchObservedRunningTime="2026-11-02 21:52:02.785388183 +0000 UTC m=+2615.224962906" Nov 02 21:52:02.970869 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:02.970753 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:03.451248 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:03.451208 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:03.455641 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:03.455241 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 02 21:52:03.455641 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:03.455328 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" podUID="d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 02 21:52:03.492196 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:03.492166 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:03.493772 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:03.493751 4935 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 21:52:03.493932 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:03.493913 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" podUID="7caa0671-29e2-468b-9dc2-d125ecd7ba84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 21:52:04.451638 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:04.451596 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 02 21:52:04.452391 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:04.452343 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" podUID="d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 02 21:52:04.491716 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:04.491677 4935 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 21:52:04.492008 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:04.491979 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" podUID="7caa0671-29e2-468b-9dc2-d125ecd7ba84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 21:52:05.336890 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:05.336805 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:05.451586 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:05.451546 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 02 21:52:05.451897 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:05.451869 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" podUID="d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 02 21:52:05.492263 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:05.492233 4935 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 21:52:05.492495 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:05.492474 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" podUID="7caa0671-29e2-468b-9dc2-d125ecd7ba84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 21:52:06.452156 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:06.451273 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 02 21:52:06.452156 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:06.451353 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" podUID="d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 02 21:52:06.492264 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:06.492225 4935 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 21:52:06.492449 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:06.492275 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" podUID="7caa0671-29e2-468b-9dc2-d125ecd7ba84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 21:52:07.451792 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:07.451737 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 02 21:52:07.452151 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:07.452083 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" podUID="d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 02 21:52:07.491968 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:07.491922 4935 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 21:52:07.492499 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:07.492464 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" podUID="7caa0671-29e2-468b-9dc2-d125ecd7ba84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 21:52:08.448969 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:08.448931 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 02 21:52:08.449331 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:08.449252 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" podUID="d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 02 21:52:08.492292 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:08.492250 4935 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 21:52:08.492816 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:08.492318 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" podUID="7caa0671-29e2-468b-9dc2-d125ecd7ba84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 21:52:09.496704 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:09.496659 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:09.497404 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:09.497378 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:09.498261 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:09.498238 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:52:09.509282 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:09.507969 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:09.509282 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:09.509039 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:09.509808 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:09.509791 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:52:10.772290 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:10.772250 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:15.953287 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:15.953244 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:16.886829 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:16.886796 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:20.955089 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:20.955037 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:25.864904 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:25.864860 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:26.085748 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:26.085706 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:28.469848 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:52:28.469810 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:52:31.138667 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:31.138617 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:33.177305 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:33.177270 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:33.453627 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:33.453534 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:36.264413 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:36.264366 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:41.646706 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:41.646609 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:46.731481 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:46.731441 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:46.959372 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:46.959337 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:47.818108 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:47.818076 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp"] Nov 02 21:52:47.823868 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:47.823846 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:47.873184 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:47.872693 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-97995b-openshift-default-dockercfg-6c7hm\"" Nov 02 21:52:47.898875 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:47.898843 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vsvr\" (UniqueName: \"kubernetes.io/projected/2d7d3443-7c43-4244-beda-b09241df09cd-kube-api-access-5vsvr\") pod \"e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:47.899061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:47.898951 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-workload-certs\") pod \"e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:47.899061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:47.899046 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d7d3443-7c43-4244-beda-b09241df09cd-istio-podinfo\") pod \"e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:47.899214 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:47.899194 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-credential-socket\") pod \"e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:47.899276 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:47.899235 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-istio-envoy\") pod \"e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:47.899276 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:47.899263 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d7d3443-7c43-4244-beda-b09241df09cd-istiod-ca-cert\") pod \"e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:47.899361 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:47.899297 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-istio-data\") pod \"e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:47.899399 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:47.899363 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d7d3443-7c43-4244-beda-b09241df09cd-istio-token\") pod \"e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:47.899438 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:47.899411 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-workload-socket\") pod \"e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:47.922260 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:47.922228 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp"] Nov 02 21:52:47.994672 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:47.994637 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:48.000328 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:48.000291 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d7d3443-7c43-4244-beda-b09241df09cd-istio-podinfo\") pod \"e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:48.000507 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:48.000391 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-credential-socket\") pod \"e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:48.000507 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:48.000419 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-istio-envoy\") pod \"e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:48.000507 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:48.000435 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d7d3443-7c43-4244-beda-b09241df09cd-istiod-ca-cert\") pod \"e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:48.000686 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:48.000548 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-istio-data\") pod \"e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:48.000686 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:48.000602 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d7d3443-7c43-4244-beda-b09241df09cd-istio-token\") pod \"e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:48.001454 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:48.000803 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-workload-socket\") pod \"e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:48.001454 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:48.000922 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vsvr\" (UniqueName: \"kubernetes.io/projected/2d7d3443-7c43-4244-beda-b09241df09cd-kube-api-access-5vsvr\") pod \"e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:48.001454 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:48.000999 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-workload-certs\") pod \"e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:48.002539 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:48.002508 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-istio-data\") pod \"e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:48.002817 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:48.002792 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-workload-socket\") pod \"e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:48.002906 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:48.002852 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-workload-certs\") pod \"e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:48.005075 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:48.005046 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d7d3443-7c43-4244-beda-b09241df09cd-istiod-ca-cert\") pod \"e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:48.005238 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:48.005067 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-istio-envoy\") pod \"e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:48.005238 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:48.005208 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-credential-socket\") pod \"e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:48.007771 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:48.007753 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d7d3443-7c43-4244-beda-b09241df09cd-istio-podinfo\") pod \"e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:48.054609 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:48.054570 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d7d3443-7c43-4244-beda-b09241df09cd-istio-token\") pod \"e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:48.054915 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:48.054891 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vsvr\" (UniqueName: \"kubernetes.io/projected/2d7d3443-7c43-4244-beda-b09241df09cd-kube-api-access-5vsvr\") pod \"e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:48.142496 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:48.142390 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:48.576655 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:48.576620 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp"] Nov 02 21:52:48.578324 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:52:48.578293 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7d3443_7c43_4244_beda_b09241df09cd.slice/crio-6a4612cf4269507a5825c6dc39488cea90dfd8924cc367b0f84ab76012340515 WatchSource:0}: Error finding container 6a4612cf4269507a5825c6dc39488cea90dfd8924cc367b0f84ab76012340515: Status 404 returned error can't find the container with id 6a4612cf4269507a5825c6dc39488cea90dfd8924cc367b0f84ab76012340515 Nov 02 21:52:48.580936 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:48.580903 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:52:48.581022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:48.580971 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:52:48.581022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:48.581003 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:52:48.953949 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:48.953857 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" event={"ID":"2d7d3443-7c43-4244-beda-b09241df09cd","Type":"ContainerStarted","Data":"da7298f3e6a8dd1e559526f3c0842554316628379f11cdf8a37951ec0e9d91d4"} Nov 02 21:52:48.953949 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:48.953913 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" event={"ID":"2d7d3443-7c43-4244-beda-b09241df09cd","Type":"ContainerStarted","Data":"6a4612cf4269507a5825c6dc39488cea90dfd8924cc367b0f84ab76012340515"} Nov 02 21:52:49.115858 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:49.115804 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" podStartSLOduration=2.1157887459999998 podStartE2EDuration="2.115788746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:52:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:52:49.10870791 +0000 UTC m=+2661.548282632" watchObservedRunningTime="2026-11-02 21:52:49.115788746 +0000 UTC m=+2661.555363491" Nov 02 21:52:49.142790 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:49.142729 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:49.148846 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:49.148819 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:49.960262 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:49.960238 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:49.961277 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:49.961251 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:52:50.087391 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:50.087355 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:51.855528 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:51.855497 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:52.142319 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:52.142231 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:55.089328 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:55.089297 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:55.320458 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:55.320427 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:57.572326 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:57.572295 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:57.779192 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:57.779160 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:58.482823 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:58.482762 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:52:58.487215 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:52:58.487176 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:52:58.764037 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:52:58.764003 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:00.566947 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:00.566902 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:01.202527 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.202490 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:01.555014 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.554972 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp"] Nov 02 21:53:01.559113 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.559082 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.601453 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.601418 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf"] Nov 02 21:53:01.605421 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.605396 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz"] Nov 02 21:53:01.605544 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.605512 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.609415 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.609390 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:01.647707 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.647677 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp"] Nov 02 21:53:01.654345 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.654307 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf"] Nov 02 21:53:01.657260 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.657231 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:01.657771 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.657749 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d6cb38-openshift-default-dockercfg-mbhbl\"" Nov 02 21:53:01.659351 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.659324 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.659533 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.659512 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.659740 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.659725 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.659879 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.659860 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.659956 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.659899 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.659956 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.659945 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nznb7\" (UniqueName: \"kubernetes.io/projected/2149054b-53f3-48ff-b5fd-dd36c73a57ca-kube-api-access-nznb7\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.660077 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.660020 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.660077 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.660055 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.660212 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.660164 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.667390 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.667368 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz"] Nov 02 21:53:01.761527 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.761493 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.761527 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.761536 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.761811 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.761570 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nznb7\" (UniqueName: \"kubernetes.io/projected/2149054b-53f3-48ff-b5fd-dd36c73a57ca-kube-api-access-nznb7\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.761811 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.761594 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.761811 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.761614 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.761811 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.761631 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:01.761811 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.761649 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.761811 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.761730 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.762169 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.761885 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.762169 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.761934 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.762169 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.761967 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkxnr\" (UniqueName: \"kubernetes.io/projected/c95c7714-4882-48d5-a3d9-c2e4313d5b52-kube-api-access-qkxnr\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:01.762169 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.761988 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.762169 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.762026 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.762169 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.762079 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:01.762169 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.762107 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.762551 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.762223 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:01.762551 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.762325 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.762551 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.762374 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.762551 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.762391 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.762551 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.762424 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:01.762551 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.762455 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:01.762893 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.762553 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:01.762893 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.762626 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4jw7\" (UniqueName: \"kubernetes.io/projected/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-kube-api-access-m4jw7\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.762893 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.762663 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.762893 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.762684 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.762893 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.762737 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:01.762893 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.762792 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.762893 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.762835 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.762893 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.762868 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.763370 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.762901 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:01.763370 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.762937 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.763370 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.763288 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.764341 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.764311 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.764341 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.764333 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.814979 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.814907 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.814979 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.814925 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nznb7\" (UniqueName: \"kubernetes.io/projected/2149054b-53f3-48ff-b5fd-dd36c73a57ca-kube-api-access-nznb7\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.864493 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.864454 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4jw7\" (UniqueName: \"kubernetes.io/projected/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-kube-api-access-m4jw7\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.864681 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.864516 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.864741 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.864693 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:01.864804 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.864746 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.864804 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.864791 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.864981 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.864822 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:01.864981 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.864894 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.864981 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.864920 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.864981 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.864923 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:01.864981 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.864973 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.865299 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.865017 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.865299 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.865062 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkxnr\" (UniqueName: \"kubernetes.io/projected/c95c7714-4882-48d5-a3d9-c2e4313d5b52-kube-api-access-qkxnr\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:01.865299 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.865110 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-workload-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:01.865299 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.865151 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.865299 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.865119 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.865299 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.865223 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:01.865299 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.865256 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.865299 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.865287 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:01.865717 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.865391 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:01.865717 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.865423 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:01.865717 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.865428 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.865717 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.865454 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.865717 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.865465 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:01.865717 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.865224 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-credential-socket\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:01.865717 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.865651 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.866118 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.865824 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istio-data\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:01.866118 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.865906 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-workload-certs\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:01.866430 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.866408 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istiod-ca-cert\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:01.867846 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.867820 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istio-envoy\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:01.867974 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.867954 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.868071 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.868045 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.868366 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.868348 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istio-podinfo\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:01.896343 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.896310 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:01.918687 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.918656 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istio-token\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:01.918825 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.918802 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkxnr\" (UniqueName: \"kubernetes.io/projected/c95c7714-4882-48d5-a3d9-c2e4313d5b52-kube-api-access-qkxnr\") pod \"e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:01.920172 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.919195 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.920172 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.919335 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4jw7\" (UniqueName: \"kubernetes.io/projected/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-kube-api-access-m4jw7\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.923663 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.923641 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:01.933483 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:01.932451 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:02.335239 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:02.335216 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp"] Nov 02 21:53:02.341834 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:02.341805 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf"] Nov 02 21:53:02.344110 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:53:02.344071 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2149054b_53f3_48ff_b5fd_dd36c73a57ca.slice/crio-f2fd22dd6b5e83e7e586da58dd18d1277672d9a64b9a884af89c2c2170e94154 WatchSource:0}: Error finding container f2fd22dd6b5e83e7e586da58dd18d1277672d9a64b9a884af89c2c2170e94154: Status 404 returned error can't find the container with id f2fd22dd6b5e83e7e586da58dd18d1277672d9a64b9a884af89c2c2170e94154 Nov 02 21:53:02.346888 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:02.346853 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:02.346996 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:02.346937 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:02.347061 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:02.347007 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:02.385763 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:53:02.385609 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff8af0f8_a8c2_4bdd_a173_604383cd3bce.slice/crio-26ff566fb29d7e9cf1e03c95aaa88febb1b1b4fe09f04bc28e83bae504fc5e6e WatchSource:0}: Error finding container 26ff566fb29d7e9cf1e03c95aaa88febb1b1b4fe09f04bc28e83bae504fc5e6e: Status 404 returned error can't find the container with id 26ff566fb29d7e9cf1e03c95aaa88febb1b1b4fe09f04bc28e83bae504fc5e6e Nov 02 21:53:02.388489 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:02.388452 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:02.388570 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:02.388537 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:02.388618 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:02.388581 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:02.388805 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:02.388786 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz"] Nov 02 21:53:02.423028 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:53:02.423002 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc95c7714_4882_48d5_a3d9_c2e4313d5b52.slice/crio-d17c65a9a4e1ccc230073928b9cf3e9649bc7c81169be1e33c2dd494044f6713 WatchSource:0}: Error finding container d17c65a9a4e1ccc230073928b9cf3e9649bc7c81169be1e33c2dd494044f6713: Status 404 returned error can't find the container with id d17c65a9a4e1ccc230073928b9cf3e9649bc7c81169be1e33c2dd494044f6713 Nov 02 21:53:02.426023 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:02.425992 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:02.426114 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:02.426080 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:02.426192 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:02.426122 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:03.049649 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:03.049606 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" event={"ID":"ff8af0f8-a8c2-4bdd-a173-604383cd3bce","Type":"ContainerStarted","Data":"93f3169f2f654928753e0146f62746258a32a6fa83104d193eb014cefa37b771"} Nov 02 21:53:03.050257 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:03.049674 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" event={"ID":"ff8af0f8-a8c2-4bdd-a173-604383cd3bce","Type":"ContainerStarted","Data":"26ff566fb29d7e9cf1e03c95aaa88febb1b1b4fe09f04bc28e83bae504fc5e6e"} Nov 02 21:53:03.051492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:03.051463 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" event={"ID":"2149054b-53f3-48ff-b5fd-dd36c73a57ca","Type":"ContainerStarted","Data":"afdf64ec5d6f25834e00254e7db2e37c173ce7d93a487d612d3ae65d4cc90e10"} Nov 02 21:53:03.051635 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:03.051516 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" event={"ID":"2149054b-53f3-48ff-b5fd-dd36c73a57ca","Type":"ContainerStarted","Data":"f2fd22dd6b5e83e7e586da58dd18d1277672d9a64b9a884af89c2c2170e94154"} Nov 02 21:53:03.053429 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:03.053403 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" event={"ID":"c95c7714-4882-48d5-a3d9-c2e4313d5b52","Type":"ContainerStarted","Data":"8cb0be668676728e16a08a7002ed859d525cd90b4f259eeb37c5ce1976caf1da"} Nov 02 21:53:03.053549 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:03.053450 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" event={"ID":"c95c7714-4882-48d5-a3d9-c2e4313d5b52","Type":"ContainerStarted","Data":"d17c65a9a4e1ccc230073928b9cf3e9649bc7c81169be1e33c2dd494044f6713"} Nov 02 21:53:03.106466 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:03.106436 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:03.165453 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:03.165393 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" podStartSLOduration=2.165379138 podStartE2EDuration="2.165379138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:53:03.161970015 +0000 UTC m=+2675.601544740" watchObservedRunningTime="2026-11-02 21:53:03.165379138 +0000 UTC m=+2675.604953860" Nov 02 21:53:03.271910 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:03.271854 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" podStartSLOduration=2.271836084 podStartE2EDuration="2.271836084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:53:03.270113506 +0000 UTC m=+2675.709688223" watchObservedRunningTime="2026-11-02 21:53:03.271836084 +0000 UTC m=+2675.711410803" Nov 02 21:53:03.463020 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:03.462783 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:03.697157 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:03.695161 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:03.897329 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:03.897294 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:03.899804 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:03.899772 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 21:53:03.900059 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:03.900034 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" podUID="2149054b-53f3-48ff-b5fd-dd36c73a57ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 21:53:03.949172 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:03.937883 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:03.949172 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:03.938086 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 21:53:03.949172 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:03.938155 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" podUID="ff8af0f8-a8c2-4bdd-a173-604383cd3bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 21:53:03.949767 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:03.949737 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:03.958510 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:03.958469 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 21:53:03.958744 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:03.958716 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" podUID="c95c7714-4882-48d5-a3d9-c2e4313d5b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 21:53:04.904589 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:04.904549 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 21:53:04.905619 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:04.905223 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" podUID="2149054b-53f3-48ff-b5fd-dd36c73a57ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 21:53:04.930938 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:04.930901 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 21:53:04.931197 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:04.931167 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" podUID="ff8af0f8-a8c2-4bdd-a173-604383cd3bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 21:53:04.943326 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:04.943294 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 21:53:04.943527 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:04.943501 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" podUID="c95c7714-4882-48d5-a3d9-c2e4313d5b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 21:53:05.898539 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:05.898497 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 21:53:05.898857 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:05.898827 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" podUID="2149054b-53f3-48ff-b5fd-dd36c73a57ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 21:53:05.939546 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:05.939501 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 21:53:05.940167 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:05.940107 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" podUID="ff8af0f8-a8c2-4bdd-a173-604383cd3bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 21:53:05.949730 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:05.949700 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 21:53:05.950370 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:05.950340 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" podUID="c95c7714-4882-48d5-a3d9-c2e4313d5b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 21:53:06.477155 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:06.474633 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:06.728221 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:06.717347 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:06.998832 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:06.998740 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:07.022232 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:07.014261 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 21:53:07.022232 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:07.014322 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" podUID="2149054b-53f3-48ff-b5fd-dd36c73a57ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 21:53:07.029176 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:07.022809 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 21:53:07.029176 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:07.022879 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" podUID="ff8af0f8-a8c2-4bdd-a173-604383cd3bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 21:53:07.029176 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:07.023189 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 21:53:07.029176 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:07.023224 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" podUID="c95c7714-4882-48d5-a3d9-c2e4313d5b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 21:53:07.262085 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:07.261976 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:07.673210 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:07.672965 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" podUID="aab5db02-4890-45e1-bcf8-1c43c335a393" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:53:07.717155 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:07.716527 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" podUID="7ba3a2e4-a47f-425a-8c65-5813ef0939c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:53:07.909158 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:07.900570 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 21:53:07.909158 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:07.900643 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" podUID="2149054b-53f3-48ff-b5fd-dd36c73a57ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 21:53:07.944211 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:07.930512 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 21:53:07.944510 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:07.944470 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" podUID="ff8af0f8-a8c2-4bdd-a173-604383cd3bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 21:53:07.944697 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:07.935437 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 21:53:07.944832 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:07.944812 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" podUID="c95c7714-4882-48d5-a3d9-c2e4313d5b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 21:53:08.898936 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:08.898503 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 21:53:08.898936 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:08.898579 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" podUID="2149054b-53f3-48ff-b5fd-dd36c73a57ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 21:53:08.933834 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:08.933784 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 21:53:08.934146 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:08.934089 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" podUID="ff8af0f8-a8c2-4bdd-a173-604383cd3bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 21:53:08.934650 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:08.934615 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 21:53:08.934808 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:08.934788 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" podUID="c95c7714-4882-48d5-a3d9-c2e4313d5b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 21:53:09.593505 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:09.593463 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:09.897228 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:09.897111 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 21:53:09.897228 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:09.897202 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" podUID="2149054b-53f3-48ff-b5fd-dd36c73a57ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 21:53:09.924280 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:09.924232 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 21:53:09.924784 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:09.924290 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" podUID="ff8af0f8-a8c2-4bdd-a173-604383cd3bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 21:53:09.933675 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:09.933640 4935 patch_prober.go:29] interesting pod/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 02 21:53:09.933856 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:09.933701 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" podUID="c95c7714-4882-48d5-a3d9-c2e4313d5b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 02 21:53:11.016891 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:11.016847 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:11.018370 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:11.018344 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:11.019878 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:11.019861 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:53:11.193042 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:11.192966 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" podStartSLOduration=10.192944914 podStartE2EDuration="10.192944914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:53:03.382967054 +0000 UTC m=+2675.822541780" watchObservedRunningTime="2026-11-02 21:53:11.192944914 +0000 UTC m=+2683.632519633" Nov 02 21:53:11.313812 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:11.313676 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:11.314702 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:11.314672 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:11.316661 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:11.316610 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:53:11.352529 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:11.352494 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:11.353818 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:11.353791 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:11.357547 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:11.357312 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:53:11.520348 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:11.520311 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:11.742824 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:11.742738 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:13.876311 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:13.876265 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" podUID="39b34814-d9a5-4505-9906-cd18e55eed4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 02 21:53:13.927316 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:13.927268 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" podUID="095af893-e150-43b9-b479-263a270d05c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 21:53:13.938225 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:13.938174 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" podUID="4534e64e-1398-44a8-acb0-22db48ebea02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:53:14.099339 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:14.099283 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" podUID="7cb3ac21-f737-4fc5-927e-4a8388582812" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:53:14.366325 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:14.366269 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" podUID="3920de75-9c3e-4826-8de5-22fcc6d50b0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:53:14.612278 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:14.612225 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" podUID="1d6a5790-dd1f-4721-bbde-3aa6b14999b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:53:14.674359 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:14.674251 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" podUID="aab5db02-4890-45e1-bcf8-1c43c335a393" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:53:14.697154 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:14.696258 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" podUID="7ba3a2e4-a47f-425a-8c65-5813ef0939c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:53:15.887949 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:15.887910 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:17.891620 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:17.891585 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:21.249639 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:21.249556 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:22.922738 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:22.922706 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:26.339064 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:26.338981 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:27.508760 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:27.508714 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:28.288377 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:28.288348 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 21:53:28.305946 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:28.305914 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 21:53:28.308672 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:28.308652 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 21:53:28.318874 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:28.318855 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 21:53:28.470296 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:53:28.470259 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:53:30.696646 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:30.696613 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:31.734810 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:31.734739 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:31.778004 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:31.777964 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv"] Nov 02 21:53:31.781804 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:31.781781 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:31.878750 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:31.878702 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv"] Nov 02 21:53:31.942629 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:31.942590 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:31.942849 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:31.942645 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dde36ed-99bc-4703-832b-b01cf6a351f1-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:31.942849 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:31.942770 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:31.942849 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:31.942804 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dde36ed-99bc-4703-832b-b01cf6a351f1-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:31.943017 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:31.942973 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:31.943077 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:31.943041 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:31.943147 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:31.943100 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dde36ed-99bc-4703-832b-b01cf6a351f1-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:31.943249 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:31.943220 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:31.943393 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:31.943335 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krs2h\" (UniqueName: \"kubernetes.io/projected/5dde36ed-99bc-4703-832b-b01cf6a351f1-kube-api-access-krs2h\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:32.044501 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:32.044461 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krs2h\" (UniqueName: \"kubernetes.io/projected/5dde36ed-99bc-4703-832b-b01cf6a351f1-kube-api-access-krs2h\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:32.044697 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:32.044547 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:32.044697 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:32.044587 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dde36ed-99bc-4703-832b-b01cf6a351f1-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:32.044697 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:32.044622 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:32.044697 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:32.044646 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dde36ed-99bc-4703-832b-b01cf6a351f1-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:32.044967 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:32.044791 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:32.044967 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:32.044849 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:32.044967 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:32.044887 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dde36ed-99bc-4703-832b-b01cf6a351f1-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:32.044967 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:32.044947 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:32.045210 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:32.045027 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-credential-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:32.045271 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:32.045225 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-workload-certs\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:32.045374 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:32.045349 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-istio-data\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:32.045587 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:32.045559 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dde36ed-99bc-4703-832b-b01cf6a351f1-istiod-ca-cert\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:32.045713 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:32.045613 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-workload-socket\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:32.047633 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:32.047608 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-istio-envoy\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:32.047755 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:32.047670 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dde36ed-99bc-4703-832b-b01cf6a351f1-istio-podinfo\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:32.097528 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:32.097493 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dde36ed-99bc-4703-832b-b01cf6a351f1-istio-token\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:32.097908 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:32.097890 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krs2h\" (UniqueName: \"kubernetes.io/projected/5dde36ed-99bc-4703-832b-b01cf6a351f1-kube-api-access-krs2h\") pod \"e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:32.395743 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:32.395665 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:32.489090 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:32.489057 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:32.818077 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:32.818054 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv"] Nov 02 21:53:32.820576 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:53:32.820545 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dde36ed_99bc_4703_832b_b01cf6a351f1.slice/crio-c425abbb30e6484de23b0aa1b6e20cc7e41e16e843344b715a66cb6500431b83 WatchSource:0}: Error finding container c425abbb30e6484de23b0aa1b6e20cc7e41e16e843344b715a66cb6500431b83: Status 404 returned error can't find the container with id c425abbb30e6484de23b0aa1b6e20cc7e41e16e843344b715a66cb6500431b83 Nov 02 21:53:32.823159 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:32.823100 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:32.823240 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:32.823182 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:32.823240 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:32.823219 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:53:33.530471 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:33.530430 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" event={"ID":"5dde36ed-99bc-4703-832b-b01cf6a351f1","Type":"ContainerStarted","Data":"01253559c47b305309d281fff1938811a93114c552dff8dd49ce9325e42705c0"} Nov 02 21:53:33.530702 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:33.530504 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" event={"ID":"5dde36ed-99bc-4703-832b-b01cf6a351f1","Type":"ContainerStarted","Data":"c425abbb30e6484de23b0aa1b6e20cc7e41e16e843344b715a66cb6500431b83"} Nov 02 21:53:33.690898 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:33.690841 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" podStartSLOduration=2.690826006 podStartE2EDuration="2.690826006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:53:33.688853974 +0000 UTC m=+2706.128428706" watchObservedRunningTime="2026-11-02 21:53:33.690826006 +0000 UTC m=+2706.130400728" Nov 02 21:53:34.395940 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:34.395903 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:34.397598 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:34.397566 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 21:53:34.397737 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:34.397631 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" podUID="5dde36ed-99bc-4703-832b-b01cf6a351f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 21:53:35.396832 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:35.396796 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 21:53:35.397304 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:35.396851 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" podUID="5dde36ed-99bc-4703-832b-b01cf6a351f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 21:53:36.396860 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:36.396824 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 21:53:36.397297 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:36.396886 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" podUID="5dde36ed-99bc-4703-832b-b01cf6a351f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 21:53:36.615654 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:36.615619 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:36.889954 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:36.889915 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:37.396436 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:37.396397 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 21:53:37.396658 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:37.396463 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" podUID="5dde36ed-99bc-4703-832b-b01cf6a351f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 21:53:37.557150 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:37.557097 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:37.780361 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:37.780309 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:38.396016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:38.395972 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 21:53:38.396249 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:38.396036 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" podUID="5dde36ed-99bc-4703-832b-b01cf6a351f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 21:53:39.396845 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:39.396810 4935 patch_prober.go:29] interesting pod/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 21:53:39.397269 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:39.396861 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" podUID="5dde36ed-99bc-4703-832b-b01cf6a351f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 21:53:40.400606 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:40.400576 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:40.401175 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:40.401083 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:40.401896 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:40.401876 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:53:42.153005 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:42.152962 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:42.390589 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:42.390546 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:46.786102 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:46.786065 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644"] Nov 02 21:53:46.791442 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:46.791420 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644" Nov 02 21:53:46.840635 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:46.839982 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-97995b-serving-cert\"" Nov 02 21:53:46.844628 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:46.844596 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644"] Nov 02 21:53:46.874121 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:46.874086 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee1633c8-eaae-419f-8912-e506e14bd98d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644\" (UID: \"ee1633c8-eaae-419f-8912-e506e14bd98d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644" Nov 02 21:53:46.874326 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:46.874224 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee1633c8-eaae-419f-8912-e506e14bd98d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644\" (UID: \"ee1633c8-eaae-419f-8912-e506e14bd98d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644" Nov 02 21:53:46.874326 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:46.874299 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqrh4\" (UniqueName: \"kubernetes.io/projected/ee1633c8-eaae-419f-8912-e506e14bd98d-kube-api-access-hqrh4\") pod \"maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644\" (UID: \"ee1633c8-eaae-419f-8912-e506e14bd98d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644" Nov 02 21:53:46.874532 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:46.874456 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee1633c8-eaae-419f-8912-e506e14bd98d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644\" (UID: \"ee1633c8-eaae-419f-8912-e506e14bd98d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644" Nov 02 21:53:46.976241 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:46.976208 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee1633c8-eaae-419f-8912-e506e14bd98d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644\" (UID: \"ee1633c8-eaae-419f-8912-e506e14bd98d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644" Nov 02 21:53:46.976433 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:46.976256 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqrh4\" (UniqueName: \"kubernetes.io/projected/ee1633c8-eaae-419f-8912-e506e14bd98d-kube-api-access-hqrh4\") pod \"maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644\" (UID: \"ee1633c8-eaae-419f-8912-e506e14bd98d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644" Nov 02 21:53:46.976433 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:53:46.976359 4935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-serving-cert: secret "maas-api-e2e-shared-a-w3-97995b-serving-cert" not found Nov 02 21:53:46.976433 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:46.976403 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee1633c8-eaae-419f-8912-e506e14bd98d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644\" (UID: \"ee1633c8-eaae-419f-8912-e506e14bd98d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644" Nov 02 21:53:46.976433 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:53:46.976420 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee1633c8-eaae-419f-8912-e506e14bd98d-maas-api-tls podName:ee1633c8-eaae-419f-8912-e506e14bd98d nodeName:}" failed. No retries permitted until 2026-11-02 21:53:47.476405492 +0000 UTC m=+2719.915980211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ee1633c8-eaae-419f-8912-e506e14bd98d-maas-api-tls") pod "maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644" (UID: "ee1633c8-eaae-419f-8912-e506e14bd98d") : secret "maas-api-e2e-shared-a-w3-97995b-serving-cert" not found Nov 02 21:53:46.976667 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:46.976522 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee1633c8-eaae-419f-8912-e506e14bd98d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644\" (UID: \"ee1633c8-eaae-419f-8912-e506e14bd98d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644" Nov 02 21:53:46.977046 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:46.977023 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee1633c8-eaae-419f-8912-e506e14bd98d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644\" (UID: \"ee1633c8-eaae-419f-8912-e506e14bd98d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644" Nov 02 21:53:46.979001 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:46.978972 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee1633c8-eaae-419f-8912-e506e14bd98d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644\" (UID: \"ee1633c8-eaae-419f-8912-e506e14bd98d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644" Nov 02 21:53:47.031945 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:47.031913 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqrh4\" (UniqueName: \"kubernetes.io/projected/ee1633c8-eaae-419f-8912-e506e14bd98d-kube-api-access-hqrh4\") pod \"maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644\" (UID: \"ee1633c8-eaae-419f-8912-e506e14bd98d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644" Nov 02 21:53:47.085823 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:47.085720 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:47.346567 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:47.346464 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:47.483278 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:47.483246 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee1633c8-eaae-419f-8912-e506e14bd98d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644\" (UID: \"ee1633c8-eaae-419f-8912-e506e14bd98d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644" Nov 02 21:53:47.485666 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:47.485647 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee1633c8-eaae-419f-8912-e506e14bd98d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644\" (UID: \"ee1633c8-eaae-419f-8912-e506e14bd98d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644" Nov 02 21:53:47.570758 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:47.570726 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:47.706723 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:47.706637 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644" Nov 02 21:53:48.083791 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:48.083762 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644"] Nov 02 21:53:48.085957 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:53:48.085921 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee1633c8_eaae_419f_8912_e506e14bd98d.slice/crio-02fa30c6f7720ae1d6b29aaea6cf4c032af08fb11dc294f9cd557a8f2395137d WatchSource:0}: Error finding container 02fa30c6f7720ae1d6b29aaea6cf4c032af08fb11dc294f9cd557a8f2395137d: Status 404 returned error can't find the container with id 02fa30c6f7720ae1d6b29aaea6cf4c032af08fb11dc294f9cd557a8f2395137d Nov 02 21:53:48.087594 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:48.087574 4935 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:53:48.653566 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:48.653526 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644" event={"ID":"ee1633c8-eaae-419f-8912-e506e14bd98d","Type":"ContainerStarted","Data":"02fa30c6f7720ae1d6b29aaea6cf4c032af08fb11dc294f9cd557a8f2395137d"} Nov 02 21:53:49.663840 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:49.663801 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644" event={"ID":"ee1633c8-eaae-419f-8912-e506e14bd98d","Type":"ContainerStarted","Data":"3b5fc5a904bbb95a6ec2957153e1c061b66d78113342208eeb6dbc796b663dbe"} Nov 02 21:53:49.664374 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:49.663875 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644" Nov 02 21:53:49.812933 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:49.812873 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644" podStartSLOduration=2.917132686 podStartE2EDuration="3.812858661s" totalImagesPullingTime="895.725975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:53:49.769917881 +0000 UTC m=+2722.209492603" watchObservedRunningTime="2026-11-02 21:53:49.812858661 +0000 UTC m=+2722.252433382" Nov 02 21:53:51.177974 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:51.177915 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:51.721416 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:51.721363 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:51.956533 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:51.956496 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:52.248763 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:52.248729 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:52.471831 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:52.471795 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:55.724351 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:55.724319 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644" Nov 02 21:53:57.993894 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:57.993859 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:58.222712 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:53:58.222676 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:53:58.470064 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:53:58.470020 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:54:02.675282 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:02.675245 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp"] Nov 02 21:54:02.679527 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:02.679504 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp" Nov 02 21:54:02.728011 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:02.727946 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-6e535d-serving-cert\"" Nov 02 21:54:02.779654 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:02.779616 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp"] Nov 02 21:54:02.793689 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:02.793654 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58c2190d-dfa6-4bcd-8469-edd252023956-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp\" (UID: \"58c2190d-dfa6-4bcd-8469-edd252023956\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp" Nov 02 21:54:02.793846 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:02.793789 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhfbg\" (UniqueName: \"kubernetes.io/projected/58c2190d-dfa6-4bcd-8469-edd252023956-kube-api-access-zhfbg\") pod \"maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp\" (UID: \"58c2190d-dfa6-4bcd-8469-edd252023956\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp" Nov 02 21:54:02.793965 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:02.793938 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58c2190d-dfa6-4bcd-8469-edd252023956-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp\" (UID: \"58c2190d-dfa6-4bcd-8469-edd252023956\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp" Nov 02 21:54:02.794186 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:02.794164 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58c2190d-dfa6-4bcd-8469-edd252023956-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp\" (UID: \"58c2190d-dfa6-4bcd-8469-edd252023956\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp" Nov 02 21:54:02.895031 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:02.894986 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58c2190d-dfa6-4bcd-8469-edd252023956-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp\" (UID: \"58c2190d-dfa6-4bcd-8469-edd252023956\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp" Nov 02 21:54:02.895229 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:02.895122 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58c2190d-dfa6-4bcd-8469-edd252023956-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp\" (UID: \"58c2190d-dfa6-4bcd-8469-edd252023956\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp" Nov 02 21:54:02.895229 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:54:02.895152 4935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-serving-cert: secret "maas-api-e2e-shared-a-w0-6e535d-serving-cert" not found Nov 02 21:54:02.895229 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:02.895190 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58c2190d-dfa6-4bcd-8469-edd252023956-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp\" (UID: \"58c2190d-dfa6-4bcd-8469-edd252023956\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp" Nov 02 21:54:02.895357 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:54:02.895230 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58c2190d-dfa6-4bcd-8469-edd252023956-maas-api-tls podName:58c2190d-dfa6-4bcd-8469-edd252023956 nodeName:}" failed. No retries permitted until 2026-11-02 21:54:03.395213394 +0000 UTC m=+2735.834788094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/58c2190d-dfa6-4bcd-8469-edd252023956-maas-api-tls") pod "maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp" (UID: "58c2190d-dfa6-4bcd-8469-edd252023956") : secret "maas-api-e2e-shared-a-w0-6e535d-serving-cert" not found Nov 02 21:54:02.895357 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:02.895278 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhfbg\" (UniqueName: \"kubernetes.io/projected/58c2190d-dfa6-4bcd-8469-edd252023956-kube-api-access-zhfbg\") pod \"maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp\" (UID: \"58c2190d-dfa6-4bcd-8469-edd252023956\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp" Nov 02 21:54:02.895861 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:02.895837 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58c2190d-dfa6-4bcd-8469-edd252023956-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp\" (UID: \"58c2190d-dfa6-4bcd-8469-edd252023956\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp" Nov 02 21:54:02.897583 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:02.897562 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58c2190d-dfa6-4bcd-8469-edd252023956-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp\" (UID: \"58c2190d-dfa6-4bcd-8469-edd252023956\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp" Nov 02 21:54:02.958193 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:02.958107 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhfbg\" (UniqueName: \"kubernetes.io/projected/58c2190d-dfa6-4bcd-8469-edd252023956-kube-api-access-zhfbg\") pod \"maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp\" (UID: \"58c2190d-dfa6-4bcd-8469-edd252023956\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp" Nov 02 21:54:02.972463 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:02.972018 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:03.198629 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:03.198586 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:03.401414 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:03.401384 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58c2190d-dfa6-4bcd-8469-edd252023956-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp\" (UID: \"58c2190d-dfa6-4bcd-8469-edd252023956\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp" Nov 02 21:54:03.404282 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:03.404252 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58c2190d-dfa6-4bcd-8469-edd252023956-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp\" (UID: \"58c2190d-dfa6-4bcd-8469-edd252023956\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp" Nov 02 21:54:03.421539 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:03.421510 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:03.592218 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:03.592188 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp" Nov 02 21:54:03.657829 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:03.657764 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:03.970412 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:03.970384 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp"] Nov 02 21:54:03.972626 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:54:03.972583 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c2190d_dfa6_4bcd_8469_edd252023956.slice/crio-ebf944d7619f4187180f3b428cf00f41fb78ecf440c312ba70450905dd7b3765 WatchSource:0}: Error finding container ebf944d7619f4187180f3b428cf00f41fb78ecf440c312ba70450905dd7b3765: Status 404 returned error can't find the container with id ebf944d7619f4187180f3b428cf00f41fb78ecf440c312ba70450905dd7b3765 Nov 02 21:54:04.829416 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:04.829380 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp" event={"ID":"58c2190d-dfa6-4bcd-8469-edd252023956","Type":"ContainerStarted","Data":"ebf944d7619f4187180f3b428cf00f41fb78ecf440c312ba70450905dd7b3765"} Nov 02 21:54:05.838904 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:05.838864 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp" event={"ID":"58c2190d-dfa6-4bcd-8469-edd252023956","Type":"ContainerStarted","Data":"0a2168fa40f1e58d73336fc5325d3fd7991c6fa0d35fbf5dec7d35da65faabed"} Nov 02 21:54:05.839454 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:05.838977 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp" Nov 02 21:54:05.949321 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:05.949262 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp" podStartSLOduration=3.066887014 podStartE2EDuration="3.94924492s" totalImagesPullingTime="882.357906ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:54:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:54:05.945748742 +0000 UTC m=+2738.385323466" watchObservedRunningTime="2026-11-02 21:54:05.94924492 +0000 UTC m=+2738.388819642" Nov 02 21:54:07.507090 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:07.507047 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:07.726290 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:07.726256 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:09.021435 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:09.021359 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft"] Nov 02 21:54:09.048481 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:09.048447 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft" Nov 02 21:54:09.077677 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:09.077647 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft"] Nov 02 21:54:09.120016 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:09.119982 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-7cde84-serving-cert\"" Nov 02 21:54:09.150157 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:09.145547 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbpp8\" (UniqueName: \"kubernetes.io/projected/dde2493b-74dc-411e-a991-cb7f91bda173-kube-api-access-jbpp8\") pod \"maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft\" (UID: \"dde2493b-74dc-411e-a991-cb7f91bda173\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft" Nov 02 21:54:09.150157 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:09.145601 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dde2493b-74dc-411e-a991-cb7f91bda173-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft\" (UID: \"dde2493b-74dc-411e-a991-cb7f91bda173\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft" Nov 02 21:54:09.150157 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:09.145763 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dde2493b-74dc-411e-a991-cb7f91bda173-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft\" (UID: \"dde2493b-74dc-411e-a991-cb7f91bda173\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft" Nov 02 21:54:09.150157 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:09.145817 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dde2493b-74dc-411e-a991-cb7f91bda173-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft\" (UID: \"dde2493b-74dc-411e-a991-cb7f91bda173\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft" Nov 02 21:54:09.250381 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:09.250335 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbpp8\" (UniqueName: \"kubernetes.io/projected/dde2493b-74dc-411e-a991-cb7f91bda173-kube-api-access-jbpp8\") pod \"maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft\" (UID: \"dde2493b-74dc-411e-a991-cb7f91bda173\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft" Nov 02 21:54:09.250688 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:09.250645 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dde2493b-74dc-411e-a991-cb7f91bda173-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft\" (UID: \"dde2493b-74dc-411e-a991-cb7f91bda173\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft" Nov 02 21:54:09.250891 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:09.250874 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dde2493b-74dc-411e-a991-cb7f91bda173-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft\" (UID: \"dde2493b-74dc-411e-a991-cb7f91bda173\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft" Nov 02 21:54:09.250980 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:54:09.250888 4935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-serving-cert: secret "maas-api-e2e-shared-a-w4-7cde84-serving-cert" not found Nov 02 21:54:09.250980 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:09.250937 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dde2493b-74dc-411e-a991-cb7f91bda173-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft\" (UID: \"dde2493b-74dc-411e-a991-cb7f91bda173\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft" Nov 02 21:54:09.250980 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:54:09.250962 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dde2493b-74dc-411e-a991-cb7f91bda173-maas-api-tls podName:dde2493b-74dc-411e-a991-cb7f91bda173 nodeName:}" failed. No retries permitted until 2026-11-02 21:54:09.750941044 +0000 UTC m=+2742.190515747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dde2493b-74dc-411e-a991-cb7f91bda173-maas-api-tls") pod "maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft" (UID: "dde2493b-74dc-411e-a991-cb7f91bda173") : secret "maas-api-e2e-shared-a-w4-7cde84-serving-cert" not found Nov 02 21:54:09.251682 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:09.251661 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dde2493b-74dc-411e-a991-cb7f91bda173-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft\" (UID: \"dde2493b-74dc-411e-a991-cb7f91bda173\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft" Nov 02 21:54:09.254381 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:09.254356 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dde2493b-74dc-411e-a991-cb7f91bda173-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft\" (UID: \"dde2493b-74dc-411e-a991-cb7f91bda173\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft" Nov 02 21:54:09.307722 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:09.307645 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbpp8\" (UniqueName: \"kubernetes.io/projected/dde2493b-74dc-411e-a991-cb7f91bda173-kube-api-access-jbpp8\") pod \"maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft\" (UID: \"dde2493b-74dc-411e-a991-cb7f91bda173\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft" Nov 02 21:54:09.316838 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:09.316807 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:09.548282 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:09.547759 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:09.758231 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:09.758107 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dde2493b-74dc-411e-a991-cb7f91bda173-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft\" (UID: \"dde2493b-74dc-411e-a991-cb7f91bda173\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft" Nov 02 21:54:09.760811 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:09.760784 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dde2493b-74dc-411e-a991-cb7f91bda173-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft\" (UID: \"dde2493b-74dc-411e-a991-cb7f91bda173\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft" Nov 02 21:54:09.766610 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:09.766578 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:09.962159 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:09.962105 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft" Nov 02 21:54:10.352913 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:10.352879 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft"] Nov 02 21:54:10.354456 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:54:10.354426 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddde2493b_74dc_411e_a991_cb7f91bda173.slice/crio-f1d74f7f1f38f841db4009b15951c28f3156f172fbddb0f99ec210a2cee59483 WatchSource:0}: Error finding container f1d74f7f1f38f841db4009b15951c28f3156f172fbddb0f99ec210a2cee59483: Status 404 returned error can't find the container with id f1d74f7f1f38f841db4009b15951c28f3156f172fbddb0f99ec210a2cee59483 Nov 02 21:54:10.878294 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:10.878248 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft" event={"ID":"dde2493b-74dc-411e-a991-cb7f91bda173","Type":"ContainerStarted","Data":"f1d74f7f1f38f841db4009b15951c28f3156f172fbddb0f99ec210a2cee59483"} Nov 02 21:54:11.855104 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:11.855075 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp" Nov 02 21:54:11.887314 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:11.887277 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft" event={"ID":"dde2493b-74dc-411e-a991-cb7f91bda173","Type":"ContainerStarted","Data":"215581e968ce16830fdc98ece279bb647a6a028e9750c5fb249756b0c1450032"} Nov 02 21:54:11.887484 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:11.887393 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft" Nov 02 21:54:12.073008 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:12.072956 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft" podStartSLOduration=3.17476589 podStartE2EDuration="4.072938098s" totalImagesPullingTime="898.172208ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:54:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:54:12.070228122 +0000 UTC m=+2744.509802836" watchObservedRunningTime="2026-11-02 21:54:12.072938098 +0000 UTC m=+2744.512512874" Nov 02 21:54:13.574175 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:13.574115 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:13.794222 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:13.794190 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:14.756188 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:14.756085 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:15.876007 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:15.875949 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:16.357333 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:16.357290 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:17.899597 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:17.899568 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft" Nov 02 21:54:18.154871 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:18.154796 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:18.940893 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:18.940855 4935 generic.go:356] "Generic (PLEG): container finished" podID="2d7d3443-7c43-4244-beda-b09241df09cd" containerID="da7298f3e6a8dd1e559526f3c0842554316628379f11cdf8a37951ec0e9d91d4" exitCode=0 Nov 02 21:54:18.941421 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:18.940930 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" event={"ID":"2d7d3443-7c43-4244-beda-b09241df09cd","Type":"ContainerDied","Data":"da7298f3e6a8dd1e559526f3c0842554316628379f11cdf8a37951ec0e9d91d4"} Nov 02 21:54:18.941421 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:18.941379 4935 scope.go:122] "RemoveContainer" containerID="da7298f3e6a8dd1e559526f3c0842554316628379f11cdf8a37951ec0e9d91d4" Nov 02 21:54:18.943009 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:18.942982 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:54:18.943102 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:18.943052 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:54:18.943102 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:18.943085 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:54:19.957212 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:19.957039 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" event={"ID":"2d7d3443-7c43-4244-beda-b09241df09cd","Type":"ContainerStarted","Data":"a7c1e97b68de8bf2fb9ea25af29723853435b4fadf8733be7467298737b6252f"} Nov 02 21:54:19.960667 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:19.960636 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:54:20.144395 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:20.144310 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:54:20.146357 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:20.146329 4935 patch_prober.go:29] interesting pod/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 21:54:20.146607 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:20.146583 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" podUID="2d7d3443-7c43-4244-beda-b09241df09cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 21:54:20.273212 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:20.264838 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:21.143579 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:21.143541 4935 patch_prober.go:29] interesting pod/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 21:54:21.144213 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:21.144178 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" podUID="2d7d3443-7c43-4244-beda-b09241df09cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 21:54:22.149167 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:22.148680 4935 patch_prober.go:29] interesting pod/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 21:54:22.149167 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:22.148752 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" podUID="2d7d3443-7c43-4244-beda-b09241df09cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 21:54:23.122374 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:23.122340 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:23.146154 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:23.145478 4935 patch_prober.go:29] interesting pod/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 21:54:23.146154 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:23.145537 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" podUID="2d7d3443-7c43-4244-beda-b09241df09cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 21:54:24.143518 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:24.143477 4935 patch_prober.go:29] interesting pod/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 21:54:24.144117 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:24.144083 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" podUID="2d7d3443-7c43-4244-beda-b09241df09cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 21:54:24.463165 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:24.463030 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:24.681767 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:24.681733 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:25.187111 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:25.185655 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:54:25.187587 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:25.187290 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 21:54:25.246757 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:25.246715 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:27.095030 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:27.094993 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:28.470679 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:54:28.470634 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:54:28.874014 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:28.873973 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:32.714701 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:32.714665 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:32.931366 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:32.931328 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:37.750467 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:37.750408 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:42.142365 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:42.142316 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:43.947161 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:43.947047 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:44.512926 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:44.512881 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:44.731963 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:44.731930 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:46.602971 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:46.602924 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:47.116613 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:47.116572 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:47.866039 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:47.866001 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:49.382955 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:49.382915 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:53.463752 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:53.463714 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:54.110173 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:54.110121 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:54.334107 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:54.334064 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:54.564451 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:54.564396 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:54.790097 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:54.790056 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:55.018118 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:55.018085 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:55.258389 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:55.258302 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:55.481240 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:55.481147 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:56.268541 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:56.268496 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:58.469980 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:54:58.469942 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:54:59.128193 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:59.128159 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:54:59.353659 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:54:59.353626 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:00.618466 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:00.618426 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:01.089550 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:01.089500 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:01.583547 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:01.583506 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:02.119596 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:02.119557 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:02.360450 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:02.360416 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:04.001607 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:04.001573 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:04.222536 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:04.222501 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:05.280077 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:05.280045 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:05.704827 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:05.704741 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:06.294413 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:06.294378 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:06.725620 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:06.725520 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:06.948367 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:06.948325 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:07.372355 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:07.372213 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:10.617392 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:10.617355 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9"] Nov 02 21:55:10.617866 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:10.617594 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" podUID="8c71c82d-68ca-434b-ae75-6b9630d5cb46" containerName="istio-proxy" containerID="cri-o://668a88949c3d6186014c9760e7c7e6a8bbc30c5bae4b92571027f5e841d84dd6" gracePeriod=30 Nov 02 21:55:10.937430 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:10.937328 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:11.157150 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:11.157103 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:12.817946 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:12.817908 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:13.170914 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:13.170833 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:13.414255 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:13.414222 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:13.870428 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:13.870375 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:14.205376 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:14.205283 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:14.439099 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:14.439062 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:15.875882 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:15.875855 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:55:15.900341 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:15.900264 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istio-token\" (UniqueName: \"kubernetes.io/projected/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istio-token\") pod \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " Nov 02 21:55:15.900341 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:15.900325 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c71c82d-68ca-434b-ae75-6b9630d5cb46-kube-api-access-mtcxj\" (UniqueName: \"kubernetes.io/projected/8c71c82d-68ca-434b-ae75-6b9630d5cb46-kube-api-access-mtcxj\") pod \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " Nov 02 21:55:15.900549 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:15.900368 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-workload-certs\") pod \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " Nov 02 21:55:15.900549 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:15.900389 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-workload-socket\") pod \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " Nov 02 21:55:15.900549 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:15.900417 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istiod-ca-cert\") pod \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " Nov 02 21:55:15.900549 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:15.900453 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-credential-socket\") pod \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " Nov 02 21:55:15.900549 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:15.900543 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istio-envoy\") pod \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " Nov 02 21:55:15.900796 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:15.900601 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istio-podinfo\") pod \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " Nov 02 21:55:15.900796 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:15.900621 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istio-data\") pod \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\" (UID: \"8c71c82d-68ca-434b-ae75-6b9630d5cb46\") " Nov 02 21:55:15.901115 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:15.901066 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-credential-socket" pod "8c71c82d-68ca-434b-ae75-6b9630d5cb46" (UID: "8c71c82d-68ca-434b-ae75-6b9630d5cb46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:55:15.901115 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:15.901082 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istio-data" pod "8c71c82d-68ca-434b-ae75-6b9630d5cb46" (UID: "8c71c82d-68ca-434b-ae75-6b9630d5cb46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:55:15.901323 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:15.901188 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-workload-certs" pod "8c71c82d-68ca-434b-ae75-6b9630d5cb46" (UID: "8c71c82d-68ca-434b-ae75-6b9630d5cb46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:55:15.901389 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:15.901326 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-workload-socket" pod "8c71c82d-68ca-434b-ae75-6b9630d5cb46" (UID: "8c71c82d-68ca-434b-ae75-6b9630d5cb46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:55:15.901650 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:15.901603 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istiod-ca-cert" pod "8c71c82d-68ca-434b-ae75-6b9630d5cb46" (UID: "8c71c82d-68ca-434b-ae75-6b9630d5cb46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:55:15.902753 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:15.902732 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istio-token" pod "8c71c82d-68ca-434b-ae75-6b9630d5cb46" (UID: "8c71c82d-68ca-434b-ae75-6b9630d5cb46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:55:15.902852 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:15.902768 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c71c82d-68ca-434b-ae75-6b9630d5cb46-kube-api-access-mtcxj" pod "8c71c82d-68ca-434b-ae75-6b9630d5cb46" (UID: "8c71c82d-68ca-434b-ae75-6b9630d5cb46"). InnerVolumeSpecName "kube-api-access-mtcxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:55:15.903086 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:15.903063 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istio-envoy" pod "8c71c82d-68ca-434b-ae75-6b9630d5cb46" (UID: "8c71c82d-68ca-434b-ae75-6b9630d5cb46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:55:15.903309 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:15.903289 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istio-podinfo" pod "8c71c82d-68ca-434b-ae75-6b9630d5cb46" (UID: "8c71c82d-68ca-434b-ae75-6b9630d5cb46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:55:16.002503 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:16.002456 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:55:16.002503 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:16.002496 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtcxj\" (UniqueName: \"kubernetes.io/projected/8c71c82d-68ca-434b-ae75-6b9630d5cb46-kube-api-access-mtcxj\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:55:16.002503 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:16.002509 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:55:16.002799 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:16.002523 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:55:16.002799 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:16.002537 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:55:16.002799 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:16.002550 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:55:16.002799 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:16.002561 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:55:16.002799 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:16.002577 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:55:16.002799 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:16.002589 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c71c82d-68ca-434b-ae75-6b9630d5cb46-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:55:16.459956 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:16.459859 4935 generic.go:356] "Generic (PLEG): container finished" podID="8c71c82d-68ca-434b-ae75-6b9630d5cb46" containerID="668a88949c3d6186014c9760e7c7e6a8bbc30c5bae4b92571027f5e841d84dd6" exitCode=0 Nov 02 21:55:16.459956 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:16.459923 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" Nov 02 21:55:16.460353 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:16.460098 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" event={"ID":"8c71c82d-68ca-434b-ae75-6b9630d5cb46","Type":"ContainerDied","Data":"668a88949c3d6186014c9760e7c7e6a8bbc30c5bae4b92571027f5e841d84dd6"} Nov 02 21:55:16.460353 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:16.460184 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9" event={"ID":"8c71c82d-68ca-434b-ae75-6b9630d5cb46","Type":"ContainerDied","Data":"17289505c687190604b2159c328a71c5945f734b284425d6d287df98c88a4618"} Nov 02 21:55:16.460353 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:16.460215 4935 scope.go:122] "RemoveContainer" containerID="668a88949c3d6186014c9760e7c7e6a8bbc30c5bae4b92571027f5e841d84dd6" Nov 02 21:55:16.484324 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:16.484019 4935 scope.go:122] "RemoveContainer" containerID="668a88949c3d6186014c9760e7c7e6a8bbc30c5bae4b92571027f5e841d84dd6" Nov 02 21:55:16.484511 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:55:16.484456 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"668a88949c3d6186014c9760e7c7e6a8bbc30c5bae4b92571027f5e841d84dd6\": container with ID starting with 668a88949c3d6186014c9760e7c7e6a8bbc30c5bae4b92571027f5e841d84dd6 not found: ID does not exist" containerID="668a88949c3d6186014c9760e7c7e6a8bbc30c5bae4b92571027f5e841d84dd6" Nov 02 21:55:16.484511 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:16.484498 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"668a88949c3d6186014c9760e7c7e6a8bbc30c5bae4b92571027f5e841d84dd6"} err="failed to get container status \"668a88949c3d6186014c9760e7c7e6a8bbc30c5bae4b92571027f5e841d84dd6\": rpc error: code = NotFound desc = could not find container \"668a88949c3d6186014c9760e7c7e6a8bbc30c5bae4b92571027f5e841d84dd6\": container with ID starting with 668a88949c3d6186014c9760e7c7e6a8bbc30c5bae4b92571027f5e841d84dd6 not found: ID does not exist" Nov 02 21:55:16.616714 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:16.616675 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9"] Nov 02 21:55:16.662635 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:16.662597 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-gljp9"] Nov 02 21:55:17.777765 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:17.777728 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c71c82d-68ca-434b-ae75-6b9630d5cb46" path="/var/lib/kubelet/pods/8c71c82d-68ca-434b-ae75-6b9630d5cb46/volumes" Nov 02 21:55:18.030332 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:18.030248 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:18.277290 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:18.277243 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:19.918519 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:19.918418 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:22.286593 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:22.286522 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:22.506367 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:22.506320 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:25.241104 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:25.241068 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:26.899166 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:26.899119 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:27.133073 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:27.133036 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:28.469449 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:55:28.469406 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:55:28.535775 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:28.535732 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:29.103510 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:29.103477 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:29.351228 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:29.351184 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:29.960678 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:29.960552 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:30.228632 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:30.228549 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:30.468760 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:30.468728 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:30.696559 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:30.696520 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:30.978010 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:30.977882 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:31.249259 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:31.249177 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:33.173905 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:33.173870 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:33.395394 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:33.395355 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:34.262969 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:34.262932 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:34.493464 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:34.493428 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:35.270524 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:35.270489 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:35.507037 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:35.507001 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:36.054443 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:36.054409 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:38.406968 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:38.406890 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:38.659387 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:38.659303 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:41.572650 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:41.572618 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:41.990167 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:41.990058 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:42.214318 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:42.214283 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:47.186549 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:47.186508 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:52.163399 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:52.163362 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:52.390256 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:52.390217 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:52.608540 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:52.608505 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:57.388470 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:55:57.388431 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:55:58.469597 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:55:58.469556 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:56:02.429170 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:02.429095 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:07.389406 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:07.389361 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:07.608807 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:07.608770 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:12.687807 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:12.687769 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:17.309868 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:17.309833 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr"] Nov 02 21:56:17.310538 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:17.310516 4935 cpu_manager.go:414] "containerMap: removing container" podUID="8c71c82d-68ca-434b-ae75-6b9630d5cb46" containerName="istio-proxy" Nov 02 21:56:17.310605 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:17.310540 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c71c82d-68ca-434b-ae75-6b9630d5cb46" containerName="istio-proxy" Nov 02 21:56:17.310652 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:17.310642 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c71c82d-68ca-434b-ae75-6b9630d5cb46" containerName="istio-proxy" Nov 02 21:56:17.314184 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:17.314162 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr" Nov 02 21:56:17.411268 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:17.411237 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-58ca7f-serving-cert\"" Nov 02 21:56:17.414438 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:17.414418 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr"] Nov 02 21:56:17.448077 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:17.448039 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1644e4fb-94a2-414c-9c35-99abbc8fe2b2-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr\" (UID: \"1644e4fb-94a2-414c-9c35-99abbc8fe2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr" Nov 02 21:56:17.448252 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:17.448211 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1644e4fb-94a2-414c-9c35-99abbc8fe2b2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr\" (UID: \"1644e4fb-94a2-414c-9c35-99abbc8fe2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr" Nov 02 21:56:17.448313 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:17.448251 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hqr6\" (UniqueName: \"kubernetes.io/projected/1644e4fb-94a2-414c-9c35-99abbc8fe2b2-kube-api-access-4hqr6\") pod \"maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr\" (UID: \"1644e4fb-94a2-414c-9c35-99abbc8fe2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr" Nov 02 21:56:17.448313 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:17.448306 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1644e4fb-94a2-414c-9c35-99abbc8fe2b2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr\" (UID: \"1644e4fb-94a2-414c-9c35-99abbc8fe2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr" Nov 02 21:56:17.549446 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:17.549407 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1644e4fb-94a2-414c-9c35-99abbc8fe2b2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr\" (UID: \"1644e4fb-94a2-414c-9c35-99abbc8fe2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr" Nov 02 21:56:17.549446 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:17.549451 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hqr6\" (UniqueName: \"kubernetes.io/projected/1644e4fb-94a2-414c-9c35-99abbc8fe2b2-kube-api-access-4hqr6\") pod \"maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr\" (UID: \"1644e4fb-94a2-414c-9c35-99abbc8fe2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr" Nov 02 21:56:17.549694 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:17.549488 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1644e4fb-94a2-414c-9c35-99abbc8fe2b2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr\" (UID: \"1644e4fb-94a2-414c-9c35-99abbc8fe2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr" Nov 02 21:56:17.549694 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:17.549590 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1644e4fb-94a2-414c-9c35-99abbc8fe2b2-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr\" (UID: \"1644e4fb-94a2-414c-9c35-99abbc8fe2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr" Nov 02 21:56:17.550347 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:17.550323 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1644e4fb-94a2-414c-9c35-99abbc8fe2b2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr\" (UID: \"1644e4fb-94a2-414c-9c35-99abbc8fe2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr" Nov 02 21:56:17.551989 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:17.551968 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1644e4fb-94a2-414c-9c35-99abbc8fe2b2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr\" (UID: \"1644e4fb-94a2-414c-9c35-99abbc8fe2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr" Nov 02 21:56:17.552066 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:17.552050 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1644e4fb-94a2-414c-9c35-99abbc8fe2b2-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr\" (UID: \"1644e4fb-94a2-414c-9c35-99abbc8fe2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr" Nov 02 21:56:17.605793 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:17.605718 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hqr6\" (UniqueName: \"kubernetes.io/projected/1644e4fb-94a2-414c-9c35-99abbc8fe2b2-kube-api-access-4hqr6\") pod \"maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr\" (UID: \"1644e4fb-94a2-414c-9c35-99abbc8fe2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr" Nov 02 21:56:17.606659 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:17.606633 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:17.627388 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:17.627366 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr" Nov 02 21:56:17.835281 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:17.835241 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:18.006327 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:18.006226 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr"] Nov 02 21:56:18.926682 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:18.926651 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr" event={"ID":"1644e4fb-94a2-414c-9c35-99abbc8fe2b2","Type":"ContainerStarted","Data":"5c5a5f8d2f937d3fc2bedcfb9dc980a13973422855b86beed0b33a926ce5cf88"} Nov 02 21:56:19.935748 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:19.935649 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr" event={"ID":"1644e4fb-94a2-414c-9c35-99abbc8fe2b2","Type":"ContainerStarted","Data":"4c9483933b04bf697513a1efc9f280acb08ab35ebef40e49bc82748791750736"} Nov 02 21:56:19.936155 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:19.935776 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr" Nov 02 21:56:20.048470 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:20.048417 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr" podStartSLOduration=2.185284677 podStartE2EDuration="3.048402716s" totalImagesPullingTime="863.118039ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:20.044873911 +0000 UTC m=+2872.484448632" watchObservedRunningTime="2026-11-02 21:56:20.048402716 +0000 UTC m=+2872.487977437" Nov 02 21:56:22.867100 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:22.867060 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:25.949322 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:25.949291 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr" Nov 02 21:56:27.887734 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:27.887685 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:28.471203 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:56:28.471163 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:56:32.917501 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:32.917453 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:33.516930 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:33.516891 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:37.885918 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:37.885874 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:38.120513 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:38.120477 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:38.346949 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:38.346908 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:42.687466 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:42.687432 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj"] Nov 02 21:56:42.691700 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:42.691676 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj" Nov 02 21:56:42.741583 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:42.741555 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-4ca09f-serving-cert\"" Nov 02 21:56:42.786757 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:42.786716 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hncp7\" (UniqueName: \"kubernetes.io/projected/297af528-36c0-423d-9bb4-0cbe3bec1bd3-kube-api-access-hncp7\") pod \"maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj\" (UID: \"297af528-36c0-423d-9bb4-0cbe3bec1bd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj" Nov 02 21:56:42.786932 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:42.786781 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/297af528-36c0-423d-9bb4-0cbe3bec1bd3-maas-api-tls\") pod \"maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj\" (UID: \"297af528-36c0-423d-9bb4-0cbe3bec1bd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj" Nov 02 21:56:42.787038 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:42.787007 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/297af528-36c0-423d-9bb4-0cbe3bec1bd3-metrics-certs\") pod \"maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj\" (UID: \"297af528-36c0-423d-9bb4-0cbe3bec1bd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj" Nov 02 21:56:42.787088 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:42.787072 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/297af528-36c0-423d-9bb4-0cbe3bec1bd3-openshift-service-ca\") pod \"maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj\" (UID: \"297af528-36c0-423d-9bb4-0cbe3bec1bd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj" Nov 02 21:56:42.793258 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:42.793230 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj"] Nov 02 21:56:42.889000 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:42.888963 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hncp7\" (UniqueName: \"kubernetes.io/projected/297af528-36c0-423d-9bb4-0cbe3bec1bd3-kube-api-access-hncp7\") pod \"maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj\" (UID: \"297af528-36c0-423d-9bb4-0cbe3bec1bd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj" Nov 02 21:56:42.889190 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:42.889011 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/297af528-36c0-423d-9bb4-0cbe3bec1bd3-maas-api-tls\") pod \"maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj\" (UID: \"297af528-36c0-423d-9bb4-0cbe3bec1bd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj" Nov 02 21:56:42.889190 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:42.889094 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/297af528-36c0-423d-9bb4-0cbe3bec1bd3-metrics-certs\") pod \"maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj\" (UID: \"297af528-36c0-423d-9bb4-0cbe3bec1bd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj" Nov 02 21:56:42.889190 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:42.889121 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/297af528-36c0-423d-9bb4-0cbe3bec1bd3-openshift-service-ca\") pod \"maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj\" (UID: \"297af528-36c0-423d-9bb4-0cbe3bec1bd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj" Nov 02 21:56:42.889323 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:56:42.889201 4935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-serving-cert: secret "maas-api-e2e-explicit-4ca09f-serving-cert" not found Nov 02 21:56:42.889323 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:56:42.889309 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/297af528-36c0-423d-9bb4-0cbe3bec1bd3-maas-api-tls podName:297af528-36c0-423d-9bb4-0cbe3bec1bd3 nodeName:}" failed. No retries permitted until 2026-11-02 21:56:43.389289274 +0000 UTC m=+2895.828863987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/297af528-36c0-423d-9bb4-0cbe3bec1bd3-maas-api-tls") pod "maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj" (UID: "297af528-36c0-423d-9bb4-0cbe3bec1bd3") : secret "maas-api-e2e-explicit-4ca09f-serving-cert" not found Nov 02 21:56:42.889834 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:42.889815 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/297af528-36c0-423d-9bb4-0cbe3bec1bd3-openshift-service-ca\") pod \"maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj\" (UID: \"297af528-36c0-423d-9bb4-0cbe3bec1bd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj" Nov 02 21:56:42.891609 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:42.891591 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/297af528-36c0-423d-9bb4-0cbe3bec1bd3-metrics-certs\") pod \"maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj\" (UID: \"297af528-36c0-423d-9bb4-0cbe3bec1bd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj" Nov 02 21:56:42.942857 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:42.942777 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hncp7\" (UniqueName: \"kubernetes.io/projected/297af528-36c0-423d-9bb4-0cbe3bec1bd3-kube-api-access-hncp7\") pod \"maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj\" (UID: \"297af528-36c0-423d-9bb4-0cbe3bec1bd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj" Nov 02 21:56:42.972887 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:42.972856 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:43.189935 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:43.189897 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:43.396478 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:43.396438 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/297af528-36c0-423d-9bb4-0cbe3bec1bd3-maas-api-tls\") pod \"maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj\" (UID: \"297af528-36c0-423d-9bb4-0cbe3bec1bd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj" Nov 02 21:56:43.398806 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:43.398771 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/297af528-36c0-423d-9bb4-0cbe3bec1bd3-maas-api-tls\") pod \"maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj\" (UID: \"297af528-36c0-423d-9bb4-0cbe3bec1bd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj" Nov 02 21:56:43.608762 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:43.608733 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj" Nov 02 21:56:43.984833 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:43.984801 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj"] Nov 02 21:56:43.985883 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:56:43.985853 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod297af528_36c0_423d_9bb4_0cbe3bec1bd3.slice/crio-98cb39556d348cfedeeb19d6306024de15aefc093b7801fe53357a1ee65724e3 WatchSource:0}: Error finding container 98cb39556d348cfedeeb19d6306024de15aefc093b7801fe53357a1ee65724e3: Status 404 returned error can't find the container with id 98cb39556d348cfedeeb19d6306024de15aefc093b7801fe53357a1ee65724e3 Nov 02 21:56:44.111746 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:44.111707 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj" event={"ID":"297af528-36c0-423d-9bb4-0cbe3bec1bd3","Type":"ContainerStarted","Data":"98cb39556d348cfedeeb19d6306024de15aefc093b7801fe53357a1ee65724e3"} Nov 02 21:56:45.121921 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:45.121883 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj" event={"ID":"297af528-36c0-423d-9bb4-0cbe3bec1bd3","Type":"ContainerStarted","Data":"31089b5cbf782ff73b6e45cac548037cec16e21a98babc000cf139ba6af2ad9f"} Nov 02 21:56:45.122412 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:45.121953 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj" Nov 02 21:56:45.262997 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:45.262942 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj" podStartSLOduration=2.223103483 podStartE2EDuration="3.262925954s" totalImagesPullingTime="1.039822471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:45.260563956 +0000 UTC m=+2897.700138678" watchObservedRunningTime="2026-11-02 21:56:45.262925954 +0000 UTC m=+2897.702500676" Nov 02 21:56:48.163342 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:48.163304 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:51.134042 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:51.134010 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj" Nov 02 21:56:53.167041 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:53.167001 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:58.257812 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:56:58.257751 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:56:58.470930 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:56:58.470892 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:57:03.366901 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:57:03.366854 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:08.428606 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:57:08.428521 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:12.364449 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:57:12.364393 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:14.001083 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:57:14.001037 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:14.450914 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:57:14.450840 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:14.672304 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:57:14.672243 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:17.692869 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:57:17.692820 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:19.589055 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:57:19.589025 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:20.773043 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:57:20.773009 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:20.976777 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:57:20.976733 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:25.131777 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:57:25.131742 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:26.663660 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:57:26.663610 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:28.471891 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:57:28.471849 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:57:28.567058 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:57:28.567020 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:30.507596 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:57:30.507564 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:31.453671 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:57:31.453633 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:33.834860 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:57:33.834818 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:36.121611 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:57:36.121554 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:40.902734 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:57:40.902699 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:41.464832 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:57:41.464799 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:46.637912 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:57:46.637867 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:51.624454 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:57:51.624409 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:56.644387 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:57:56.644339 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:57:58.472652 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:57:58.472612 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:57:59.842407 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:57:59.842373 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:02.015351 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:02.015314 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:02.232765 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:02.232730 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:02.606281 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:02.606235 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:03.253300 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:03.248672 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:03.472244 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:03.472204 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:03.696593 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:03.696557 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:07.522181 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:07.522144 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:07.730791 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:07.730754 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:07.936511 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:07.936419 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:10.691499 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:10.691459 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:10.896927 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:10.896889 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:12.938446 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:12.938387 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:17.933874 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:17.933825 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:18.352224 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:18.352185 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl"] Nov 02 21:58:18.356746 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:18.356717 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" Nov 02 21:58:18.409170 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:18.409096 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl"] Nov 02 21:58:18.435992 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:18.435956 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-6e535d\"/\"kube-root-ca.crt\"" Nov 02 21:58:18.450655 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:18.450629 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-6e535d\"/\"openshift-service-ca.crt\"" Nov 02 21:58:18.450997 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:18.450979 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-6e535d\"/\"default-dockercfg-wf5bh\"" Nov 02 21:58:18.450997 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:18.450984 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-6e535d\"/\"auth-test-model-6e535d-kserve-self-signed-certs\"" Nov 02 21:58:18.482877 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:18.482847 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21877724-6647-4d7a-ae34-d01a1da09ed8-tmp-dir\") pod \"auth-test-model-6e535d-kserve-5489cb5f7-z2kfl\" (UID: \"21877724-6647-4d7a-ae34-d01a1da09ed8\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" Nov 02 21:58:18.483012 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:18.482902 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21877724-6647-4d7a-ae34-d01a1da09ed8-tls-certs\") pod \"auth-test-model-6e535d-kserve-5489cb5f7-z2kfl\" (UID: \"21877724-6647-4d7a-ae34-d01a1da09ed8\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" Nov 02 21:58:18.483066 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:18.483024 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21877724-6647-4d7a-ae34-d01a1da09ed8-home\") pod \"auth-test-model-6e535d-kserve-5489cb5f7-z2kfl\" (UID: \"21877724-6647-4d7a-ae34-d01a1da09ed8\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" Nov 02 21:58:18.483164 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:18.483124 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8sf4\" (UniqueName: \"kubernetes.io/projected/21877724-6647-4d7a-ae34-d01a1da09ed8-kube-api-access-v8sf4\") pod \"auth-test-model-6e535d-kserve-5489cb5f7-z2kfl\" (UID: \"21877724-6647-4d7a-ae34-d01a1da09ed8\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" Nov 02 21:58:18.483214 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:18.483180 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21877724-6647-4d7a-ae34-d01a1da09ed8-model-cache\") pod \"auth-test-model-6e535d-kserve-5489cb5f7-z2kfl\" (UID: \"21877724-6647-4d7a-ae34-d01a1da09ed8\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" Nov 02 21:58:18.483262 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:18.483214 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21877724-6647-4d7a-ae34-d01a1da09ed8-dshm\") pod \"auth-test-model-6e535d-kserve-5489cb5f7-z2kfl\" (UID: \"21877724-6647-4d7a-ae34-d01a1da09ed8\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" Nov 02 21:58:18.540852 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:18.540806 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:18.584455 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:18.584409 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21877724-6647-4d7a-ae34-d01a1da09ed8-tmp-dir\") pod \"auth-test-model-6e535d-kserve-5489cb5f7-z2kfl\" (UID: \"21877724-6647-4d7a-ae34-d01a1da09ed8\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" Nov 02 21:58:18.584648 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:18.584482 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21877724-6647-4d7a-ae34-d01a1da09ed8-tls-certs\") pod \"auth-test-model-6e535d-kserve-5489cb5f7-z2kfl\" (UID: \"21877724-6647-4d7a-ae34-d01a1da09ed8\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" Nov 02 21:58:18.584648 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:18.584539 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21877724-6647-4d7a-ae34-d01a1da09ed8-home\") pod \"auth-test-model-6e535d-kserve-5489cb5f7-z2kfl\" (UID: \"21877724-6647-4d7a-ae34-d01a1da09ed8\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" Nov 02 21:58:18.584648 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:18.584607 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8sf4\" (UniqueName: \"kubernetes.io/projected/21877724-6647-4d7a-ae34-d01a1da09ed8-kube-api-access-v8sf4\") pod \"auth-test-model-6e535d-kserve-5489cb5f7-z2kfl\" (UID: \"21877724-6647-4d7a-ae34-d01a1da09ed8\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" Nov 02 21:58:18.584820 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:18.584647 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21877724-6647-4d7a-ae34-d01a1da09ed8-model-cache\") pod \"auth-test-model-6e535d-kserve-5489cb5f7-z2kfl\" (UID: \"21877724-6647-4d7a-ae34-d01a1da09ed8\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" Nov 02 21:58:18.584820 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:18.584696 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21877724-6647-4d7a-ae34-d01a1da09ed8-dshm\") pod \"auth-test-model-6e535d-kserve-5489cb5f7-z2kfl\" (UID: \"21877724-6647-4d7a-ae34-d01a1da09ed8\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" Nov 02 21:58:18.584948 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:18.584916 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21877724-6647-4d7a-ae34-d01a1da09ed8-tmp-dir\") pod \"auth-test-model-6e535d-kserve-5489cb5f7-z2kfl\" (UID: \"21877724-6647-4d7a-ae34-d01a1da09ed8\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" Nov 02 21:58:18.585321 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:18.585296 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21877724-6647-4d7a-ae34-d01a1da09ed8-home\") pod \"auth-test-model-6e535d-kserve-5489cb5f7-z2kfl\" (UID: \"21877724-6647-4d7a-ae34-d01a1da09ed8\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" Nov 02 21:58:18.585436 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:18.585318 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21877724-6647-4d7a-ae34-d01a1da09ed8-model-cache\") pod \"auth-test-model-6e535d-kserve-5489cb5f7-z2kfl\" (UID: \"21877724-6647-4d7a-ae34-d01a1da09ed8\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" Nov 02 21:58:18.587222 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:18.587194 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21877724-6647-4d7a-ae34-d01a1da09ed8-tls-certs\") pod \"auth-test-model-6e535d-kserve-5489cb5f7-z2kfl\" (UID: \"21877724-6647-4d7a-ae34-d01a1da09ed8\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" Nov 02 21:58:18.587402 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:18.587380 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21877724-6647-4d7a-ae34-d01a1da09ed8-dshm\") pod \"auth-test-model-6e535d-kserve-5489cb5f7-z2kfl\" (UID: \"21877724-6647-4d7a-ae34-d01a1da09ed8\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" Nov 02 21:58:18.637922 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:18.637833 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8sf4\" (UniqueName: \"kubernetes.io/projected/21877724-6647-4d7a-ae34-d01a1da09ed8-kube-api-access-v8sf4\") pod \"auth-test-model-6e535d-kserve-5489cb5f7-z2kfl\" (UID: \"21877724-6647-4d7a-ae34-d01a1da09ed8\") " pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" Nov 02 21:58:18.671498 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:18.671450 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" Nov 02 21:58:19.057751 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:19.057724 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl"] Nov 02 21:58:19.060581 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:58:19.060542 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21877724_6647_4d7a_ae34_d01a1da09ed8.slice/crio-54998987f00ad9dce8ba837e0979f480892a73587df2e469c3d44da60cd42d80 WatchSource:0}: Error finding container 54998987f00ad9dce8ba837e0979f480892a73587df2e469c3d44da60cd42d80: Status 404 returned error can't find the container with id 54998987f00ad9dce8ba837e0979f480892a73587df2e469c3d44da60cd42d80 Nov 02 21:58:19.897968 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:19.897871 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" event={"ID":"21877724-6647-4d7a-ae34-d01a1da09ed8","Type":"ContainerStarted","Data":"823ccb1037d0caf791301dd4ed564081ae370b1b829567ae117d927157c8fdeb"} Nov 02 21:58:19.897968 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:19.897931 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" event={"ID":"21877724-6647-4d7a-ae34-d01a1da09ed8","Type":"ContainerStarted","Data":"54998987f00ad9dce8ba837e0979f480892a73587df2e469c3d44da60cd42d80"} Nov 02 21:58:20.005952 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:20.005898 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" podStartSLOduration=2.005883242 podStartE2EDuration="2.005883242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:58:20.003728871 +0000 UTC m=+2992.443303594" watchObservedRunningTime="2026-11-02 21:58:20.005883242 +0000 UTC m=+2992.445457965" Nov 02 21:58:20.672917 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:20.672879 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" Nov 02 21:58:23.017578 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:23.017544 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:27.990230 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:27.990193 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:28.344314 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:28.344286 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 21:58:28.357123 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:28.357092 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 21:58:28.364642 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:28.364615 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 21:58:28.375258 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:28.375239 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 21:58:28.469537 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:58:28.469499 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:58:28.672937 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:28.672847 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" Nov 02 21:58:28.686191 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:28.686166 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" Nov 02 21:58:28.970886 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:28.970811 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" Nov 02 21:58:30.801204 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:30.801168 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:32.968419 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:32.968380 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:33.987852 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:33.987819 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf"] Nov 02 21:58:33.988311 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:33.988034 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" podUID="ff8af0f8-a8c2-4bdd-a173-604383cd3bce" containerName="istio-proxy" containerID="cri-o://93f3169f2f654928753e0146f62746258a32a6fa83104d193eb014cefa37b771" gracePeriod=30 Nov 02 21:58:34.035265 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:34.035222 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp"] Nov 02 21:58:34.035555 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:34.035518 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" podUID="2149054b-53f3-48ff-b5fd-dd36c73a57ca" containerName="istio-proxy" containerID="cri-o://afdf64ec5d6f25834e00254e7db2e37c173ce7d93a487d612d3ae65d4cc90e10" gracePeriod=30 Nov 02 21:58:34.038626 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:34.038594 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv"] Nov 02 21:58:34.038891 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:34.038863 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" podUID="5dde36ed-99bc-4703-832b-b01cf6a351f1" containerName="istio-proxy" containerID="cri-o://01253559c47b305309d281fff1938811a93114c552dff8dd49ce9325e42705c0" gracePeriod=30 Nov 02 21:58:34.082869 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:34.082826 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd"] Nov 02 21:58:34.083179 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:34.083146 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" podUID="d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee" containerName="istio-proxy" containerID="cri-o://5aca8b81fcf8be259a0920c0f16042145ea27df2578234539a72ccc90f13b1b9" gracePeriod=30 Nov 02 21:58:34.165085 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:34.165051 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:34.375292 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:34.375253 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:38.017197 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:38.017101 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:39.344612 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.344578 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:58:39.387983 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.387951 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-credential-socket\") pod \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " Nov 02 21:58:39.388180 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.388048 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-kube-api-access-m4jw7\" (UniqueName: \"kubernetes.io/projected/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-kube-api-access-m4jw7\") pod \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " Nov 02 21:58:39.388180 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.388089 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istio-podinfo\") pod \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " Nov 02 21:58:39.388180 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.388165 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-workload-certs\") pod \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " Nov 02 21:58:39.388796 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.388196 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istio-envoy\") pod \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " Nov 02 21:58:39.388796 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.388273 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istio-token\" (UniqueName: \"kubernetes.io/projected/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istio-token\") pod \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " Nov 02 21:58:39.388796 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.388322 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istiod-ca-cert\") pod \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " Nov 02 21:58:39.388796 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.388391 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-workload-socket\") pod \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " Nov 02 21:58:39.388796 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.388451 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istio-data\") pod \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\" (UID: \"ff8af0f8-a8c2-4bdd-a173-604383cd3bce\") " Nov 02 21:58:39.388796 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.388523 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-credential-socket" pod "ff8af0f8-a8c2-4bdd-a173-604383cd3bce" (UID: "ff8af0f8-a8c2-4bdd-a173-604383cd3bce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:39.389172 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.389042 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.389901 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.389850 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-workload-socket" pod "ff8af0f8-a8c2-4bdd-a173-604383cd3bce" (UID: "ff8af0f8-a8c2-4bdd-a173-604383cd3bce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:39.390062 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.389997 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istiod-ca-cert" pod "ff8af0f8-a8c2-4bdd-a173-604383cd3bce" (UID: "ff8af0f8-a8c2-4bdd-a173-604383cd3bce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:58:39.390313 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.390289 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-workload-certs" pod "ff8af0f8-a8c2-4bdd-a173-604383cd3bce" (UID: "ff8af0f8-a8c2-4bdd-a173-604383cd3bce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:39.390787 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.390741 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istio-data" pod "ff8af0f8-a8c2-4bdd-a173-604383cd3bce" (UID: "ff8af0f8-a8c2-4bdd-a173-604383cd3bce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:39.393709 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.393666 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istio-token" pod "ff8af0f8-a8c2-4bdd-a173-604383cd3bce" (UID: "ff8af0f8-a8c2-4bdd-a173-604383cd3bce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:39.394615 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.394580 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istio-podinfo" pod "ff8af0f8-a8c2-4bdd-a173-604383cd3bce" (UID: "ff8af0f8-a8c2-4bdd-a173-604383cd3bce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:58:39.394714 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.394671 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-kube-api-access-m4jw7" pod "ff8af0f8-a8c2-4bdd-a173-604383cd3bce" (UID: "ff8af0f8-a8c2-4bdd-a173-604383cd3bce"). InnerVolumeSpecName "kube-api-access-m4jw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:39.396011 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.395980 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istio-envoy" pod "ff8af0f8-a8c2-4bdd-a173-604383cd3bce" (UID: "ff8af0f8-a8c2-4bdd-a173-604383cd3bce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:39.467576 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.467491 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:58:39.474179 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.474152 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:58:39.488236 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.488215 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:58:39.489421 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.489398 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istio-token\" (UniqueName: \"kubernetes.io/projected/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istio-token\") pod \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " Nov 02 21:58:39.489535 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.489432 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-credential-socket\") pod \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " Nov 02 21:58:39.489535 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.489454 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-kube-api-access-9vhwb\" (UniqueName: \"kubernetes.io/projected/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-kube-api-access-9vhwb\") pod \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " Nov 02 21:58:39.489535 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.489477 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istiod-ca-cert\") pod \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " Nov 02 21:58:39.489535 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.489504 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-istio-envoy\") pod \"5dde36ed-99bc-4703-832b-b01cf6a351f1\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " Nov 02 21:58:39.489535 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.489521 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-workload-socket\") pod \"5dde36ed-99bc-4703-832b-b01cf6a351f1\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " Nov 02 21:58:39.489808 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.489552 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dde36ed-99bc-4703-832b-b01cf6a351f1-kube-api-access-krs2h\" (UniqueName: \"kubernetes.io/projected/5dde36ed-99bc-4703-832b-b01cf6a351f1-kube-api-access-krs2h\") pod \"5dde36ed-99bc-4703-832b-b01cf6a351f1\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " Nov 02 21:58:39.489808 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.489583 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istio-data\") pod \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " Nov 02 21:58:39.489808 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.489761 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-credential-socket" pod "d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee" (UID: "d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:39.489982 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.489793 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istio-data" pod "d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee" (UID: "d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:39.489982 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.489897 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-workload-socket" pod "5dde36ed-99bc-4703-832b-b01cf6a351f1" (UID: "5dde36ed-99bc-4703-832b-b01cf6a351f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:39.489982 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.489930 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istiod-ca-cert" pod "2149054b-53f3-48ff-b5fd-dd36c73a57ca" (UID: "2149054b-53f3-48ff-b5fd-dd36c73a57ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:58:39.489982 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.489930 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-credential-socket\") pod \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " Nov 02 21:58:39.490213 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.489984 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-credential-socket\") pod \"5dde36ed-99bc-4703-832b-b01cf6a351f1\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " Nov 02 21:58:39.490213 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.490059 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2149054b-53f3-48ff-b5fd-dd36c73a57ca-kube-api-access-nznb7\" (UniqueName: \"kubernetes.io/projected/2149054b-53f3-48ff-b5fd-dd36c73a57ca-kube-api-access-nznb7\") pod \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " Nov 02 21:58:39.490213 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.490183 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-credential-socket" pod "2149054b-53f3-48ff-b5fd-dd36c73a57ca" (UID: "2149054b-53f3-48ff-b5fd-dd36c73a57ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:39.490376 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.490242 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istio-envoy\") pod \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " Nov 02 21:58:39.490376 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.490257 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-credential-socket" pod "5dde36ed-99bc-4703-832b-b01cf6a351f1" (UID: "5dde36ed-99bc-4703-832b-b01cf6a351f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:39.490376 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.490275 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-istio-data\") pod \"5dde36ed-99bc-4703-832b-b01cf6a351f1\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " Nov 02 21:58:39.490376 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.490355 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-workload-socket\") pod \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " Nov 02 21:58:39.490600 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.490428 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-workload-certs\") pod \"5dde36ed-99bc-4703-832b-b01cf6a351f1\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " Nov 02 21:58:39.490600 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.490473 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-workload-socket\") pod \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " Nov 02 21:58:39.490600 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.490533 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istiod-ca-cert\") pod \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " Nov 02 21:58:39.490600 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.490566 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istio-podinfo\") pod \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " Nov 02 21:58:39.490862 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.490612 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istio-data\") pod \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " Nov 02 21:58:39.490862 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.490641 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istio-token\" (UniqueName: \"kubernetes.io/projected/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istio-token\") pod \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " Nov 02 21:58:39.490862 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.490673 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istio-podinfo\") pod \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " Nov 02 21:58:39.490862 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.490722 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dde36ed-99bc-4703-832b-b01cf6a351f1-istio-token\" (UniqueName: \"kubernetes.io/projected/5dde36ed-99bc-4703-832b-b01cf6a351f1-istio-token\") pod \"5dde36ed-99bc-4703-832b-b01cf6a351f1\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " Nov 02 21:58:39.490862 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.490789 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5dde36ed-99bc-4703-832b-b01cf6a351f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dde36ed-99bc-4703-832b-b01cf6a351f1-istio-podinfo\") pod \"5dde36ed-99bc-4703-832b-b01cf6a351f1\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " Nov 02 21:58:39.490862 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.490827 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-workload-certs\") pod \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " Nov 02 21:58:39.491217 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.490873 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istio-envoy\") pod \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\" (UID: \"2149054b-53f3-48ff-b5fd-dd36c73a57ca\") " Nov 02 21:58:39.491217 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.490904 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dde36ed-99bc-4703-832b-b01cf6a351f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dde36ed-99bc-4703-832b-b01cf6a351f1-istiod-ca-cert\") pod \"5dde36ed-99bc-4703-832b-b01cf6a351f1\" (UID: \"5dde36ed-99bc-4703-832b-b01cf6a351f1\") " Nov 02 21:58:39.491217 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.490941 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-workload-certs\") pod \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\" (UID: \"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee\") " Nov 02 21:58:39.491642 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.491620 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.491724 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.491649 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.491724 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.491665 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.491724 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.491682 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.491724 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.491698 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.491724 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.491713 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.492271 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.491726 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4jw7\" (UniqueName: \"kubernetes.io/projected/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-kube-api-access-m4jw7\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.492271 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.491742 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.492271 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.491764 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.492271 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.491779 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.492271 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.491796 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.492271 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.491811 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff8af0f8-a8c2-4bdd-a173-604383cd3bce-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.492271 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.491828 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.492271 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.491843 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.493764 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.490608 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-istio-data" pod "5dde36ed-99bc-4703-832b-b01cf6a351f1" (UID: "5dde36ed-99bc-4703-832b-b01cf6a351f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:39.493764 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.490786 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-workload-socket" pod "d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee" (UID: "d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:39.493764 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.490941 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-workload-socket" pod "2149054b-53f3-48ff-b5fd-dd36c73a57ca" (UID: "2149054b-53f3-48ff-b5fd-dd36c73a57ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:39.493764 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.491102 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-workload-certs" pod "5dde36ed-99bc-4703-832b-b01cf6a351f1" (UID: "5dde36ed-99bc-4703-832b-b01cf6a351f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:39.493764 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.491998 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istio-token" pod "d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee" (UID: "d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:39.493764 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.492099 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-workload-certs" pod "d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee" (UID: "d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:39.493764 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.492267 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-workload-certs" pod "2149054b-53f3-48ff-b5fd-dd36c73a57ca" (UID: "2149054b-53f3-48ff-b5fd-dd36c73a57ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:39.493764 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.492282 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istio-data" pod "2149054b-53f3-48ff-b5fd-dd36c73a57ca" (UID: "2149054b-53f3-48ff-b5fd-dd36c73a57ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:39.493764 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.492746 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istiod-ca-cert" pod "d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee" (UID: "d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:58:39.493764 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.492805 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-istio-envoy" pod "5dde36ed-99bc-4703-832b-b01cf6a351f1" (UID: "5dde36ed-99bc-4703-832b-b01cf6a351f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:39.493764 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.492863 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-kube-api-access-9vhwb" pod "d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee" (UID: "d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee"). InnerVolumeSpecName "kube-api-access-9vhwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:39.493764 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.492884 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dde36ed-99bc-4703-832b-b01cf6a351f1-kube-api-access-krs2h" pod "5dde36ed-99bc-4703-832b-b01cf6a351f1" (UID: "5dde36ed-99bc-4703-832b-b01cf6a351f1"). InnerVolumeSpecName "kube-api-access-krs2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:39.493764 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.493281 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dde36ed-99bc-4703-832b-b01cf6a351f1-istiod-ca-cert" pod "5dde36ed-99bc-4703-832b-b01cf6a351f1" (UID: "5dde36ed-99bc-4703-832b-b01cf6a351f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:58:39.493764 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.493701 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2149054b-53f3-48ff-b5fd-dd36c73a57ca-kube-api-access-nznb7" pod "2149054b-53f3-48ff-b5fd-dd36c73a57ca" (UID: "2149054b-53f3-48ff-b5fd-dd36c73a57ca"). InnerVolumeSpecName "kube-api-access-nznb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:39.494628 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.493851 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istio-podinfo" pod "2149054b-53f3-48ff-b5fd-dd36c73a57ca" (UID: "2149054b-53f3-48ff-b5fd-dd36c73a57ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:58:39.495091 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.495065 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istio-envoy" pod "d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee" (UID: "d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:39.495208 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.495096 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dde36ed-99bc-4703-832b-b01cf6a351f1-istio-token" pod "5dde36ed-99bc-4703-832b-b01cf6a351f1" (UID: "5dde36ed-99bc-4703-832b-b01cf6a351f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:39.495208 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.495114 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istio-envoy" pod "2149054b-53f3-48ff-b5fd-dd36c73a57ca" (UID: "2149054b-53f3-48ff-b5fd-dd36c73a57ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:39.495208 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.495193 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5dde36ed-99bc-4703-832b-b01cf6a351f1-istio-podinfo" pod "5dde36ed-99bc-4703-832b-b01cf6a351f1" (UID: "5dde36ed-99bc-4703-832b-b01cf6a351f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:58:39.495395 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.495247 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istio-podinfo" pod "d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee" (UID: "d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:58:39.495452 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.495444 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istio-token" pod "2149054b-53f3-48ff-b5fd-dd36c73a57ca" (UID: "2149054b-53f3-48ff-b5fd-dd36c73a57ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:39.593215 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.593118 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.593215 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.593160 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.593215 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.593171 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.593215 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.593184 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.593215 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.593197 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.593215 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.593210 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.593215 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.593218 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.593590 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.593227 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.593590 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.593236 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.593590 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.593245 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.593590 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.593253 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dde36ed-99bc-4703-832b-b01cf6a351f1-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.593590 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.593264 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dde36ed-99bc-4703-832b-b01cf6a351f1-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.593590 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.593278 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.593590 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.593286 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2149054b-53f3-48ff-b5fd-dd36c73a57ca-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.593590 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.593320 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dde36ed-99bc-4703-832b-b01cf6a351f1-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.593590 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.593329 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.593590 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.593339 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.593590 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.593354 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vhwb\" (UniqueName: \"kubernetes.io/projected/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee-kube-api-access-9vhwb\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.593590 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.593366 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dde36ed-99bc-4703-832b-b01cf6a351f1-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.593590 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.593376 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krs2h\" (UniqueName: \"kubernetes.io/projected/5dde36ed-99bc-4703-832b-b01cf6a351f1-kube-api-access-krs2h\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:39.593590 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:39.593385 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nznb7\" (UniqueName: \"kubernetes.io/projected/2149054b-53f3-48ff-b5fd-dd36c73a57ca-kube-api-access-nznb7\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:40.038008 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.037971 4935 generic.go:356] "Generic (PLEG): container finished" podID="d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee" containerID="5aca8b81fcf8be259a0920c0f16042145ea27df2578234539a72ccc90f13b1b9" exitCode=0 Nov 02 21:58:40.038221 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.038038 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" Nov 02 21:58:40.038221 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.038063 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" event={"ID":"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee","Type":"ContainerDied","Data":"5aca8b81fcf8be259a0920c0f16042145ea27df2578234539a72ccc90f13b1b9"} Nov 02 21:58:40.038221 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.038161 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd" event={"ID":"d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee","Type":"ContainerDied","Data":"2ea36479975c811184dbfff0c2ce02dc3f2b33c8e9dcf4ce049e1bd37d4a2d01"} Nov 02 21:58:40.038395 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.038204 4935 scope.go:122] "RemoveContainer" containerID="5aca8b81fcf8be259a0920c0f16042145ea27df2578234539a72ccc90f13b1b9" Nov 02 21:58:40.039749 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.039723 4935 generic.go:356] "Generic (PLEG): container finished" podID="5dde36ed-99bc-4703-832b-b01cf6a351f1" containerID="01253559c47b305309d281fff1938811a93114c552dff8dd49ce9325e42705c0" exitCode=0 Nov 02 21:58:40.039909 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.039786 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" Nov 02 21:58:40.039976 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.039902 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" event={"ID":"5dde36ed-99bc-4703-832b-b01cf6a351f1","Type":"ContainerDied","Data":"01253559c47b305309d281fff1938811a93114c552dff8dd49ce9325e42705c0"} Nov 02 21:58:40.039976 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.039941 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv" event={"ID":"5dde36ed-99bc-4703-832b-b01cf6a351f1","Type":"ContainerDied","Data":"c425abbb30e6484de23b0aa1b6e20cc7e41e16e843344b715a66cb6500431b83"} Nov 02 21:58:40.041828 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.041803 4935 generic.go:356] "Generic (PLEG): container finished" podID="ff8af0f8-a8c2-4bdd-a173-604383cd3bce" containerID="93f3169f2f654928753e0146f62746258a32a6fa83104d193eb014cefa37b771" exitCode=0 Nov 02 21:58:40.042374 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.041838 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" Nov 02 21:58:40.042374 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.041883 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" event={"ID":"ff8af0f8-a8c2-4bdd-a173-604383cd3bce","Type":"ContainerDied","Data":"93f3169f2f654928753e0146f62746258a32a6fa83104d193eb014cefa37b771"} Nov 02 21:58:40.042374 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.041926 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf" event={"ID":"ff8af0f8-a8c2-4bdd-a173-604383cd3bce","Type":"ContainerDied","Data":"26ff566fb29d7e9cf1e03c95aaa88febb1b1b4fe09f04bc28e83bae504fc5e6e"} Nov 02 21:58:40.043912 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.043633 4935 generic.go:356] "Generic (PLEG): container finished" podID="2149054b-53f3-48ff-b5fd-dd36c73a57ca" containerID="afdf64ec5d6f25834e00254e7db2e37c173ce7d93a487d612d3ae65d4cc90e10" exitCode=0 Nov 02 21:58:40.043912 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.043701 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" Nov 02 21:58:40.043912 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.043754 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" event={"ID":"2149054b-53f3-48ff-b5fd-dd36c73a57ca","Type":"ContainerDied","Data":"afdf64ec5d6f25834e00254e7db2e37c173ce7d93a487d612d3ae65d4cc90e10"} Nov 02 21:58:40.043912 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.043786 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp" event={"ID":"2149054b-53f3-48ff-b5fd-dd36c73a57ca","Type":"ContainerDied","Data":"f2fd22dd6b5e83e7e586da58dd18d1277672d9a64b9a884af89c2c2170e94154"} Nov 02 21:58:40.051603 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.051585 4935 scope.go:122] "RemoveContainer" containerID="5aca8b81fcf8be259a0920c0f16042145ea27df2578234539a72ccc90f13b1b9" Nov 02 21:58:40.051891 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:58:40.051869 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aca8b81fcf8be259a0920c0f16042145ea27df2578234539a72ccc90f13b1b9\": container with ID starting with 5aca8b81fcf8be259a0920c0f16042145ea27df2578234539a72ccc90f13b1b9 not found: ID does not exist" containerID="5aca8b81fcf8be259a0920c0f16042145ea27df2578234539a72ccc90f13b1b9" Nov 02 21:58:40.051952 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.051925 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aca8b81fcf8be259a0920c0f16042145ea27df2578234539a72ccc90f13b1b9"} err="failed to get container status \"5aca8b81fcf8be259a0920c0f16042145ea27df2578234539a72ccc90f13b1b9\": rpc error: code = NotFound desc = could not find container \"5aca8b81fcf8be259a0920c0f16042145ea27df2578234539a72ccc90f13b1b9\": container with ID starting with 5aca8b81fcf8be259a0920c0f16042145ea27df2578234539a72ccc90f13b1b9 not found: ID does not exist" Nov 02 21:58:40.052003 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.051955 4935 scope.go:122] "RemoveContainer" containerID="01253559c47b305309d281fff1938811a93114c552dff8dd49ce9325e42705c0" Nov 02 21:58:40.063492 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.063469 4935 scope.go:122] "RemoveContainer" containerID="01253559c47b305309d281fff1938811a93114c552dff8dd49ce9325e42705c0" Nov 02 21:58:40.063772 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:58:40.063752 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01253559c47b305309d281fff1938811a93114c552dff8dd49ce9325e42705c0\": container with ID starting with 01253559c47b305309d281fff1938811a93114c552dff8dd49ce9325e42705c0 not found: ID does not exist" containerID="01253559c47b305309d281fff1938811a93114c552dff8dd49ce9325e42705c0" Nov 02 21:58:40.063855 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.063778 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01253559c47b305309d281fff1938811a93114c552dff8dd49ce9325e42705c0"} err="failed to get container status \"01253559c47b305309d281fff1938811a93114c552dff8dd49ce9325e42705c0\": rpc error: code = NotFound desc = could not find container \"01253559c47b305309d281fff1938811a93114c552dff8dd49ce9325e42705c0\": container with ID starting with 01253559c47b305309d281fff1938811a93114c552dff8dd49ce9325e42705c0 not found: ID does not exist" Nov 02 21:58:40.063855 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.063794 4935 scope.go:122] "RemoveContainer" containerID="93f3169f2f654928753e0146f62746258a32a6fa83104d193eb014cefa37b771" Nov 02 21:58:40.073380 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.073358 4935 scope.go:122] "RemoveContainer" containerID="93f3169f2f654928753e0146f62746258a32a6fa83104d193eb014cefa37b771" Nov 02 21:58:40.073639 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:58:40.073620 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93f3169f2f654928753e0146f62746258a32a6fa83104d193eb014cefa37b771\": container with ID starting with 93f3169f2f654928753e0146f62746258a32a6fa83104d193eb014cefa37b771 not found: ID does not exist" containerID="93f3169f2f654928753e0146f62746258a32a6fa83104d193eb014cefa37b771" Nov 02 21:58:40.073692 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.073647 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93f3169f2f654928753e0146f62746258a32a6fa83104d193eb014cefa37b771"} err="failed to get container status \"93f3169f2f654928753e0146f62746258a32a6fa83104d193eb014cefa37b771\": rpc error: code = NotFound desc = could not find container \"93f3169f2f654928753e0146f62746258a32a6fa83104d193eb014cefa37b771\": container with ID starting with 93f3169f2f654928753e0146f62746258a32a6fa83104d193eb014cefa37b771 not found: ID does not exist" Nov 02 21:58:40.073692 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.073665 4935 scope.go:122] "RemoveContainer" containerID="afdf64ec5d6f25834e00254e7db2e37c173ce7d93a487d612d3ae65d4cc90e10" Nov 02 21:58:40.082490 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.082464 4935 scope.go:122] "RemoveContainer" containerID="afdf64ec5d6f25834e00254e7db2e37c173ce7d93a487d612d3ae65d4cc90e10" Nov 02 21:58:40.082742 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:58:40.082724 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afdf64ec5d6f25834e00254e7db2e37c173ce7d93a487d612d3ae65d4cc90e10\": container with ID starting with afdf64ec5d6f25834e00254e7db2e37c173ce7d93a487d612d3ae65d4cc90e10 not found: ID does not exist" containerID="afdf64ec5d6f25834e00254e7db2e37c173ce7d93a487d612d3ae65d4cc90e10" Nov 02 21:58:40.082805 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.082753 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afdf64ec5d6f25834e00254e7db2e37c173ce7d93a487d612d3ae65d4cc90e10"} err="failed to get container status \"afdf64ec5d6f25834e00254e7db2e37c173ce7d93a487d612d3ae65d4cc90e10\": rpc error: code = NotFound desc = could not find container \"afdf64ec5d6f25834e00254e7db2e37c173ce7d93a487d612d3ae65d4cc90e10\": container with ID starting with afdf64ec5d6f25834e00254e7db2e37c173ce7d93a487d612d3ae65d4cc90e10 not found: ID does not exist" Nov 02 21:58:40.192580 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.192550 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd"] Nov 02 21:58:40.238639 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.238607 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-55xkd"] Nov 02 21:58:40.297344 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.297312 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf"] Nov 02 21:58:40.300066 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.300034 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-h2lwf"] Nov 02 21:58:40.404064 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.404030 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv"] Nov 02 21:58:40.406888 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.406866 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-78qdv"] Nov 02 21:58:40.507364 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.507330 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp"] Nov 02 21:58:40.511734 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:40.511706 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0c9014-openshift-default-558577f6f5-7k5gp"] Nov 02 21:58:41.778093 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:41.778060 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2149054b-53f3-48ff-b5fd-dd36c73a57ca" path="/var/lib/kubelet/pods/2149054b-53f3-48ff-b5fd-dd36c73a57ca/volumes" Nov 02 21:58:41.778648 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:41.778631 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dde36ed-99bc-4703-832b-b01cf6a351f1" path="/var/lib/kubelet/pods/5dde36ed-99bc-4703-832b-b01cf6a351f1/volumes" Nov 02 21:58:41.779087 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:41.779074 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee" path="/var/lib/kubelet/pods/d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee/volumes" Nov 02 21:58:41.779853 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:41.779838 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff8af0f8-a8c2-4bdd-a173-604383cd3bce" path="/var/lib/kubelet/pods/ff8af0f8-a8c2-4bdd-a173-604383cd3bce/volumes" Nov 02 21:58:43.005310 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:43.005278 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:47.195070 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.195033 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl"] Nov 02 21:58:47.195612 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.195596 4935 cpu_manager.go:414] "containerMap: removing container" podUID="5dde36ed-99bc-4703-832b-b01cf6a351f1" containerName="istio-proxy" Nov 02 21:58:47.195656 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.195613 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dde36ed-99bc-4703-832b-b01cf6a351f1" containerName="istio-proxy" Nov 02 21:58:47.195656 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.195629 4935 cpu_manager.go:414] "containerMap: removing container" podUID="ff8af0f8-a8c2-4bdd-a173-604383cd3bce" containerName="istio-proxy" Nov 02 21:58:47.195656 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.195636 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff8af0f8-a8c2-4bdd-a173-604383cd3bce" containerName="istio-proxy" Nov 02 21:58:47.195656 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.195650 4935 cpu_manager.go:414] "containerMap: removing container" podUID="2149054b-53f3-48ff-b5fd-dd36c73a57ca" containerName="istio-proxy" Nov 02 21:58:47.195656 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.195655 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2149054b-53f3-48ff-b5fd-dd36c73a57ca" containerName="istio-proxy" Nov 02 21:58:47.195810 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.195672 4935 cpu_manager.go:414] "containerMap: removing container" podUID="d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee" containerName="istio-proxy" Nov 02 21:58:47.195810 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.195678 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee" containerName="istio-proxy" Nov 02 21:58:47.195810 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.195783 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="2149054b-53f3-48ff-b5fd-dd36c73a57ca" containerName="istio-proxy" Nov 02 21:58:47.195810 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.195793 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="d064c27a-7a1c-4fe2-ad81-9db92ce9d6ee" containerName="istio-proxy" Nov 02 21:58:47.195810 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.195804 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dde36ed-99bc-4703-832b-b01cf6a351f1" containerName="istio-proxy" Nov 02 21:58:47.195810 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.195811 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff8af0f8-a8c2-4bdd-a173-604383cd3bce" containerName="istio-proxy" Nov 02 21:58:47.200077 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.200058 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" Nov 02 21:58:47.255456 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.255417 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl"] Nov 02 21:58:47.291320 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.291283 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39599ffa-7e36-4a75-b0f7-efacc85bda8f-dshm\") pod \"explicit-model-4ca09f-kserve-6488fddb7c-mzrjl\" (UID: \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\") " pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" Nov 02 21:58:47.291320 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.291330 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39599ffa-7e36-4a75-b0f7-efacc85bda8f-tls-certs\") pod \"explicit-model-4ca09f-kserve-6488fddb7c-mzrjl\" (UID: \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\") " pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" Nov 02 21:58:47.291603 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.291404 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39599ffa-7e36-4a75-b0f7-efacc85bda8f-model-cache\") pod \"explicit-model-4ca09f-kserve-6488fddb7c-mzrjl\" (UID: \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\") " pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" Nov 02 21:58:47.291603 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.291435 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92jt4\" (UniqueName: \"kubernetes.io/projected/39599ffa-7e36-4a75-b0f7-efacc85bda8f-kube-api-access-92jt4\") pod \"explicit-model-4ca09f-kserve-6488fddb7c-mzrjl\" (UID: \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\") " pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" Nov 02 21:58:47.291603 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.291465 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39599ffa-7e36-4a75-b0f7-efacc85bda8f-home\") pod \"explicit-model-4ca09f-kserve-6488fddb7c-mzrjl\" (UID: \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\") " pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" Nov 02 21:58:47.291806 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.291643 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39599ffa-7e36-4a75-b0f7-efacc85bda8f-tmp-dir\") pod \"explicit-model-4ca09f-kserve-6488fddb7c-mzrjl\" (UID: \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\") " pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" Nov 02 21:58:47.295432 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.295400 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-4ca09f\"/\"openshift-service-ca.crt\"" Nov 02 21:58:47.295432 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.295419 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-4ca09f\"/\"kube-root-ca.crt\"" Nov 02 21:58:47.304898 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.295955 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-4ca09f\"/\"explicit-model-4ca09f-kserve-self-signed-certs\"" Nov 02 21:58:47.304898 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.295975 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-4ca09f\"/\"default-dockercfg-ldq67\"" Nov 02 21:58:47.380732 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.380697 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:47.393251 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.393214 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39599ffa-7e36-4a75-b0f7-efacc85bda8f-tmp-dir\") pod \"explicit-model-4ca09f-kserve-6488fddb7c-mzrjl\" (UID: \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\") " pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" Nov 02 21:58:47.393486 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.393342 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39599ffa-7e36-4a75-b0f7-efacc85bda8f-dshm\") pod \"explicit-model-4ca09f-kserve-6488fddb7c-mzrjl\" (UID: \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\") " pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" Nov 02 21:58:47.393486 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.393385 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39599ffa-7e36-4a75-b0f7-efacc85bda8f-tls-certs\") pod \"explicit-model-4ca09f-kserve-6488fddb7c-mzrjl\" (UID: \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\") " pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" Nov 02 21:58:47.393486 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.393445 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39599ffa-7e36-4a75-b0f7-efacc85bda8f-model-cache\") pod \"explicit-model-4ca09f-kserve-6488fddb7c-mzrjl\" (UID: \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\") " pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" Nov 02 21:58:47.393486 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.393486 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92jt4\" (UniqueName: \"kubernetes.io/projected/39599ffa-7e36-4a75-b0f7-efacc85bda8f-kube-api-access-92jt4\") pod \"explicit-model-4ca09f-kserve-6488fddb7c-mzrjl\" (UID: \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\") " pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" Nov 02 21:58:47.393724 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.393521 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39599ffa-7e36-4a75-b0f7-efacc85bda8f-home\") pod \"explicit-model-4ca09f-kserve-6488fddb7c-mzrjl\" (UID: \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\") " pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" Nov 02 21:58:47.393893 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.393873 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39599ffa-7e36-4a75-b0f7-efacc85bda8f-home\") pod \"explicit-model-4ca09f-kserve-6488fddb7c-mzrjl\" (UID: \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\") " pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" Nov 02 21:58:47.394067 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.394046 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39599ffa-7e36-4a75-b0f7-efacc85bda8f-tmp-dir\") pod \"explicit-model-4ca09f-kserve-6488fddb7c-mzrjl\" (UID: \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\") " pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" Nov 02 21:58:47.394451 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.394426 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39599ffa-7e36-4a75-b0f7-efacc85bda8f-model-cache\") pod \"explicit-model-4ca09f-kserve-6488fddb7c-mzrjl\" (UID: \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\") " pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" Nov 02 21:58:47.397192 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.396758 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39599ffa-7e36-4a75-b0f7-efacc85bda8f-dshm\") pod \"explicit-model-4ca09f-kserve-6488fddb7c-mzrjl\" (UID: \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\") " pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" Nov 02 21:58:47.400212 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.398505 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39599ffa-7e36-4a75-b0f7-efacc85bda8f-tls-certs\") pod \"explicit-model-4ca09f-kserve-6488fddb7c-mzrjl\" (UID: \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\") " pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" Nov 02 21:58:47.453490 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.453413 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92jt4\" (UniqueName: \"kubernetes.io/projected/39599ffa-7e36-4a75-b0f7-efacc85bda8f-kube-api-access-92jt4\") pod \"explicit-model-4ca09f-kserve-6488fddb7c-mzrjl\" (UID: \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\") " pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" Nov 02 21:58:47.512236 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.512202 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" Nov 02 21:58:47.897093 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:47.897061 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl"] Nov 02 21:58:47.898225 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:58:47.898198 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39599ffa_7e36_4a75_b0f7_efacc85bda8f.slice/crio-287e51eb785983b891f3c49cbca851cf3786dddbf04f026a833471da47f48678 WatchSource:0}: Error finding container 287e51eb785983b891f3c49cbca851cf3786dddbf04f026a833471da47f48678: Status 404 returned error can't find the container with id 287e51eb785983b891f3c49cbca851cf3786dddbf04f026a833471da47f48678 Nov 02 21:58:48.067806 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:48.067765 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:48.114395 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:48.114358 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" event={"ID":"39599ffa-7e36-4a75-b0f7-efacc85bda8f","Type":"ContainerStarted","Data":"cc21b4f63161de008178642f9f15d6f7918de1e625ea24abe00b515ae9f1953f"} Nov 02 21:58:48.114395 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:48.114404 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" event={"ID":"39599ffa-7e36-4a75-b0f7-efacc85bda8f","Type":"ContainerStarted","Data":"287e51eb785983b891f3c49cbca851cf3786dddbf04f026a833471da47f48678"} Nov 02 21:58:48.226827 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:48.226717 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" podStartSLOduration=1.226701604 podStartE2EDuration="1.226701604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:58:48.220583745 +0000 UTC m=+3020.660158468" watchObservedRunningTime="2026-11-02 21:58:48.226701604 +0000 UTC m=+3020.666276325" Nov 02 21:58:48.512354 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:48.512320 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" Nov 02 21:58:49.045556 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:49.045514 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz"] Nov 02 21:58:49.045811 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:49.045784 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" podUID="c95c7714-4882-48d5-a3d9-c2e4313d5b52" containerName="istio-proxy" containerID="cri-o://8cb0be668676728e16a08a7002ed859d525cd90b4f259eeb37c5ce1976caf1da" gracePeriod=30 Nov 02 21:58:49.091520 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:49.091485 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p"] Nov 02 21:58:49.091751 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:49.091721 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" podUID="7caa0671-29e2-468b-9dc2-d125ecd7ba84" containerName="istio-proxy" containerID="cri-o://0a957eed5d2c2e83955d9e66dcb33663ba52edaa6c76e1119a2d5465c34ec8cf" gracePeriod=30 Nov 02 21:58:53.117711 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:53.117675 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:54.162449 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.162409 4935 generic.go:356] "Generic (PLEG): container finished" podID="c95c7714-4882-48d5-a3d9-c2e4313d5b52" containerID="8cb0be668676728e16a08a7002ed859d525cd90b4f259eeb37c5ce1976caf1da" exitCode=0 Nov 02 21:58:54.162820 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.162442 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" event={"ID":"c95c7714-4882-48d5-a3d9-c2e4313d5b52","Type":"ContainerDied","Data":"8cb0be668676728e16a08a7002ed859d525cd90b4f259eeb37c5ce1976caf1da"} Nov 02 21:58:54.361896 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.361866 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:58:54.405022 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.404999 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:58:54.484402 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.484370 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istio-podinfo\") pod \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " Nov 02 21:58:54.484536 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.484426 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7caa0671-29e2-468b-9dc2-d125ecd7ba84-kube-api-access-bfmsh\" (UniqueName: \"kubernetes.io/projected/7caa0671-29e2-468b-9dc2-d125ecd7ba84-kube-api-access-bfmsh\") pod \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " Nov 02 21:58:54.484536 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.484446 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c95c7714-4882-48d5-a3d9-c2e4313d5b52-kube-api-access-qkxnr\" (UniqueName: \"kubernetes.io/projected/c95c7714-4882-48d5-a3d9-c2e4313d5b52-kube-api-access-qkxnr\") pod \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " Nov 02 21:58:54.484652 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.484553 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istio-token\" (UniqueName: \"kubernetes.io/projected/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istio-token\") pod \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " Nov 02 21:58:54.484652 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.484607 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-workload-certs\") pod \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " Nov 02 21:58:54.484740 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.484674 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istiod-ca-cert\") pod \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " Nov 02 21:58:54.484740 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.484715 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istio-token\" (UniqueName: \"kubernetes.io/projected/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istio-token\") pod \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " Nov 02 21:58:54.484844 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.484747 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-credential-socket\") pod \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " Nov 02 21:58:54.484905 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.484871 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istio-envoy\") pod \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " Nov 02 21:58:54.484965 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.484924 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istio-envoy\") pod \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " Nov 02 21:58:54.485026 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.484947 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-workload-certs" pod "c95c7714-4882-48d5-a3d9-c2e4313d5b52" (UID: "c95c7714-4882-48d5-a3d9-c2e4313d5b52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.485026 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.484976 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istio-podinfo\") pod \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " Nov 02 21:58:54.485168 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.485052 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istio-data\") pod \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " Nov 02 21:58:54.485168 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.485117 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-workload-socket\") pod \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " Nov 02 21:58:54.485283 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.485196 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istio-data\") pod \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " Nov 02 21:58:54.485283 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.485247 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istiod-ca-cert\") pod \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " Nov 02 21:58:54.485392 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.485328 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-workload-socket\") pod \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " Nov 02 21:58:54.485453 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.485381 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-credential-socket\") pod \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\" (UID: \"c95c7714-4882-48d5-a3d9-c2e4313d5b52\") " Nov 02 21:58:54.485453 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.485437 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-workload-certs\") pod \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\" (UID: \"7caa0671-29e2-468b-9dc2-d125ecd7ba84\") " Nov 02 21:58:54.485629 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.485596 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istiod-ca-cert" pod "7caa0671-29e2-468b-9dc2-d125ecd7ba84" (UID: "7caa0671-29e2-468b-9dc2-d125ecd7ba84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:58:54.486886 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.485883 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-workload-socket" pod "7caa0671-29e2-468b-9dc2-d125ecd7ba84" (UID: "7caa0671-29e2-468b-9dc2-d125ecd7ba84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.486886 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.486159 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-workload-certs" pod "7caa0671-29e2-468b-9dc2-d125ecd7ba84" (UID: "7caa0671-29e2-468b-9dc2-d125ecd7ba84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.486886 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.486324 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istiod-ca-cert" pod "c95c7714-4882-48d5-a3d9-c2e4313d5b52" (UID: "c95c7714-4882-48d5-a3d9-c2e4313d5b52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:58:54.486886 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.486326 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-workload-socket" pod "c95c7714-4882-48d5-a3d9-c2e4313d5b52" (UID: "c95c7714-4882-48d5-a3d9-c2e4313d5b52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.486886 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.486397 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-credential-socket" pod "c95c7714-4882-48d5-a3d9-c2e4313d5b52" (UID: "c95c7714-4882-48d5-a3d9-c2e4313d5b52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.486886 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.486519 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istio-data" pod "7caa0671-29e2-468b-9dc2-d125ecd7ba84" (UID: "7caa0671-29e2-468b-9dc2-d125ecd7ba84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.486886 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.486610 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-credential-socket" pod "7caa0671-29e2-468b-9dc2-d125ecd7ba84" (UID: "7caa0671-29e2-468b-9dc2-d125ecd7ba84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.486886 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.486801 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istio-data" pod "c95c7714-4882-48d5-a3d9-c2e4313d5b52" (UID: "c95c7714-4882-48d5-a3d9-c2e4313d5b52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.487405 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.487017 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.487405 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.487033 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istio-podinfo" pod "7caa0671-29e2-468b-9dc2-d125ecd7ba84" (UID: "7caa0671-29e2-468b-9dc2-d125ecd7ba84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:58:54.487405 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.487049 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.487405 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.487064 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.487405 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.487078 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.487405 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.487093 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.487405 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.487103 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c95c7714-4882-48d5-a3d9-c2e4313d5b52-kube-api-access-qkxnr" pod "c95c7714-4882-48d5-a3d9-c2e4313d5b52" (UID: "c95c7714-4882-48d5-a3d9-c2e4313d5b52"). InnerVolumeSpecName "kube-api-access-qkxnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:54.487405 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.487106 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.487405 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.487121 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.487405 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.487148 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.487405 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.487162 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.487405 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.487176 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.488341 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.488317 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istio-token" pod "c95c7714-4882-48d5-a3d9-c2e4313d5b52" (UID: "c95c7714-4882-48d5-a3d9-c2e4313d5b52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:54.488748 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.488728 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istio-podinfo" pod "c95c7714-4882-48d5-a3d9-c2e4313d5b52" (UID: "c95c7714-4882-48d5-a3d9-c2e4313d5b52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:58:54.488825 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.488752 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istio-envoy" pod "c95c7714-4882-48d5-a3d9-c2e4313d5b52" (UID: "c95c7714-4882-48d5-a3d9-c2e4313d5b52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.488919 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.488902 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7caa0671-29e2-468b-9dc2-d125ecd7ba84-kube-api-access-bfmsh" pod "7caa0671-29e2-468b-9dc2-d125ecd7ba84" (UID: "7caa0671-29e2-468b-9dc2-d125ecd7ba84"). InnerVolumeSpecName "kube-api-access-bfmsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:54.489798 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.489781 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istio-envoy" pod "7caa0671-29e2-468b-9dc2-d125ecd7ba84" (UID: "7caa0671-29e2-468b-9dc2-d125ecd7ba84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:54.489991 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.489975 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istio-token" pod "7caa0671-29e2-468b-9dc2-d125ecd7ba84" (UID: "7caa0671-29e2-468b-9dc2-d125ecd7ba84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:54.588224 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.588116 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.588224 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.588175 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfmsh\" (UniqueName: \"kubernetes.io/projected/7caa0671-29e2-468b-9dc2-d125ecd7ba84-kube-api-access-bfmsh\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.588224 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.588187 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkxnr\" (UniqueName: \"kubernetes.io/projected/c95c7714-4882-48d5-a3d9-c2e4313d5b52-kube-api-access-qkxnr\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.588224 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.588197 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.588224 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.588208 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.588224 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.588216 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.588224 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.588229 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7caa0671-29e2-468b-9dc2-d125ecd7ba84-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:54.588700 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:54.588237 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c95c7714-4882-48d5-a3d9-c2e4313d5b52-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:55.170220 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:55.170189 4935 generic.go:356] "Generic (PLEG): container finished" podID="7caa0671-29e2-468b-9dc2-d125ecd7ba84" containerID="0a957eed5d2c2e83955d9e66dcb33663ba52edaa6c76e1119a2d5465c34ec8cf" exitCode=0 Nov 02 21:58:55.170707 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:55.170238 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" Nov 02 21:58:55.170707 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:55.170276 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" event={"ID":"7caa0671-29e2-468b-9dc2-d125ecd7ba84","Type":"ContainerDied","Data":"0a957eed5d2c2e83955d9e66dcb33663ba52edaa6c76e1119a2d5465c34ec8cf"} Nov 02 21:58:55.170707 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:55.170333 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p" event={"ID":"7caa0671-29e2-468b-9dc2-d125ecd7ba84","Type":"ContainerDied","Data":"97b9b1539dcd90f667d55b879a03bc1c71ea1cdba64cc7a63866a117ec31de52"} Nov 02 21:58:55.170707 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:55.170349 4935 scope.go:122] "RemoveContainer" containerID="0a957eed5d2c2e83955d9e66dcb33663ba52edaa6c76e1119a2d5465c34ec8cf" Nov 02 21:58:55.171837 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:55.171806 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" event={"ID":"c95c7714-4882-48d5-a3d9-c2e4313d5b52","Type":"ContainerDied","Data":"d17c65a9a4e1ccc230073928b9cf3e9649bc7c81169be1e33c2dd494044f6713"} Nov 02 21:58:55.171952 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:55.171896 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz" Nov 02 21:58:55.182430 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:55.182412 4935 scope.go:122] "RemoveContainer" containerID="0a957eed5d2c2e83955d9e66dcb33663ba52edaa6c76e1119a2d5465c34ec8cf" Nov 02 21:58:55.182708 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:58:55.182685 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a957eed5d2c2e83955d9e66dcb33663ba52edaa6c76e1119a2d5465c34ec8cf\": container with ID starting with 0a957eed5d2c2e83955d9e66dcb33663ba52edaa6c76e1119a2d5465c34ec8cf not found: ID does not exist" containerID="0a957eed5d2c2e83955d9e66dcb33663ba52edaa6c76e1119a2d5465c34ec8cf" Nov 02 21:58:55.182813 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:55.182719 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a957eed5d2c2e83955d9e66dcb33663ba52edaa6c76e1119a2d5465c34ec8cf"} err="failed to get container status \"0a957eed5d2c2e83955d9e66dcb33663ba52edaa6c76e1119a2d5465c34ec8cf\": rpc error: code = NotFound desc = could not find container \"0a957eed5d2c2e83955d9e66dcb33663ba52edaa6c76e1119a2d5465c34ec8cf\": container with ID starting with 0a957eed5d2c2e83955d9e66dcb33663ba52edaa6c76e1119a2d5465c34ec8cf not found: ID does not exist" Nov 02 21:58:55.182813 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:55.182740 4935 scope.go:122] "RemoveContainer" containerID="8cb0be668676728e16a08a7002ed859d525cd90b4f259eeb37c5ce1976caf1da" Nov 02 21:58:55.374124 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:55.374035 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz"] Nov 02 21:58:55.419091 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:55.419060 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d6cb38-openshift-default-58dff5fd97-x4bqz"] Nov 02 21:58:55.487951 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:55.487908 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p"] Nov 02 21:58:55.487951 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:55.487961 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-4kj6p"] Nov 02 21:58:55.777210 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:55.777173 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7caa0671-29e2-468b-9dc2-d125ecd7ba84" path="/var/lib/kubelet/pods/7caa0671-29e2-468b-9dc2-d125ecd7ba84/volumes" Nov 02 21:58:55.777759 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:55.777735 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c95c7714-4882-48d5-a3d9-c2e4313d5b52" path="/var/lib/kubelet/pods/c95c7714-4882-48d5-a3d9-c2e4313d5b52/volumes" Nov 02 21:58:57.513123 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:57.513086 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" Nov 02 21:58:57.525356 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:57.525333 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" Nov 02 21:58:58.116272 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:58.116239 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:58:58.204725 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:58:58.204689 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" Nov 02 21:58:58.469969 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:58:58.469887 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:59:03.103679 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:03.103641 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:08.065797 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:08.065747 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:13.028605 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:13.028567 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:17.896801 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:17.896759 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:18.162104 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.161952 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:18.165506 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.164406 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq"] Nov 02 21:59:18.165506 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.165089 4935 cpu_manager.go:414] "containerMap: removing container" podUID="7caa0671-29e2-468b-9dc2-d125ecd7ba84" containerName="istio-proxy" Nov 02 21:59:18.165506 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.165103 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7caa0671-29e2-468b-9dc2-d125ecd7ba84" containerName="istio-proxy" Nov 02 21:59:18.165506 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.165154 4935 cpu_manager.go:414] "containerMap: removing container" podUID="c95c7714-4882-48d5-a3d9-c2e4313d5b52" containerName="istio-proxy" Nov 02 21:59:18.165506 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.165163 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c95c7714-4882-48d5-a3d9-c2e4313d5b52" containerName="istio-proxy" Nov 02 21:59:18.165506 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.165292 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="c95c7714-4882-48d5-a3d9-c2e4313d5b52" containerName="istio-proxy" Nov 02 21:59:18.165506 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.165316 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="7caa0671-29e2-468b-9dc2-d125ecd7ba84" containerName="istio-proxy" Nov 02 21:59:18.168812 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.168794 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" Nov 02 21:59:18.222502 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.222462 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq"] Nov 02 21:59:18.260969 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.260927 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-dshm\") pod \"auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq\" (UID: \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" Nov 02 21:59:18.261184 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.260982 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-home\") pod \"auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq\" (UID: \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" Nov 02 21:59:18.261184 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.261086 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-tmp-dir\") pod \"auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq\" (UID: \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" Nov 02 21:59:18.261184 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.261115 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-model-cache\") pod \"auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq\" (UID: \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" Nov 02 21:59:18.261184 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.261148 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-tls-certs\") pod \"auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq\" (UID: \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" Nov 02 21:59:18.261398 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.261288 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj5lb\" (UniqueName: \"kubernetes.io/projected/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-kube-api-access-nj5lb\") pod \"auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq\" (UID: \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" Nov 02 21:59:18.261663 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.261644 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-a1d751\"/\"openshift-service-ca.crt\"" Nov 02 21:59:18.261779 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.261644 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-a1d751\"/\"default-dockercfg-vmzss\"" Nov 02 21:59:18.261779 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.261711 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-a1d751\"/\"auth-test-model-a1d751-kserve-self-signed-certs\"" Nov 02 21:59:18.264371 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.264350 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-a1d751\"/\"kube-root-ca.crt\"" Nov 02 21:59:18.362574 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.362538 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-tls-certs\") pod \"auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq\" (UID: \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" Nov 02 21:59:18.362785 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.362625 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj5lb\" (UniqueName: \"kubernetes.io/projected/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-kube-api-access-nj5lb\") pod \"auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq\" (UID: \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" Nov 02 21:59:18.362854 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.362791 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-dshm\") pod \"auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq\" (UID: \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" Nov 02 21:59:18.363300 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.363267 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-home\") pod \"auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq\" (UID: \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" Nov 02 21:59:18.363403 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.363355 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-tmp-dir\") pod \"auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq\" (UID: \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" Nov 02 21:59:18.363403 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.363385 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-model-cache\") pod \"auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq\" (UID: \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" Nov 02 21:59:18.363612 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.363588 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-home\") pod \"auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq\" (UID: \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" Nov 02 21:59:18.363765 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.363739 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-model-cache\") pod \"auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq\" (UID: \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" Nov 02 21:59:18.364009 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.363991 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-tmp-dir\") pod \"auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq\" (UID: \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" Nov 02 21:59:18.365688 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.365664 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-dshm\") pod \"auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq\" (UID: \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" Nov 02 21:59:18.366272 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.366223 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:18.372909 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.372865 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-tls-certs\") pod \"auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq\" (UID: \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" Nov 02 21:59:18.417310 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.417248 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj5lb\" (UniqueName: \"kubernetes.io/projected/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-kube-api-access-nj5lb\") pod \"auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq\" (UID: \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\") " pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" Nov 02 21:59:18.482370 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.482309 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" Nov 02 21:59:18.861841 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:18.861812 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq"] Nov 02 21:59:18.863924 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:59:18.863891 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7e3a7a2_ba1b_4dcd_b9aa_43f22f44398a.slice/crio-6b391f7322167c5e2b7663b8f6cf19f50c29cf115189594b6b25cbb19b1938ee WatchSource:0}: Error finding container 6b391f7322167c5e2b7663b8f6cf19f50c29cf115189594b6b25cbb19b1938ee: Status 404 returned error can't find the container with id 6b391f7322167c5e2b7663b8f6cf19f50c29cf115189594b6b25cbb19b1938ee Nov 02 21:59:19.339411 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:19.339373 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" event={"ID":"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a","Type":"ContainerStarted","Data":"cdc6edde1e3370cd4026533b6075a6b4069f7f59fd2fd56462771ce8b8100638"} Nov 02 21:59:19.339411 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:19.339422 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" event={"ID":"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a","Type":"ContainerStarted","Data":"6b391f7322167c5e2b7663b8f6cf19f50c29cf115189594b6b25cbb19b1938ee"} Nov 02 21:59:19.447145 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:19.447077 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" podStartSLOduration=1.447060975 podStartE2EDuration="1.447060975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:59:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:59:19.443845336 +0000 UTC m=+3051.883420058" watchObservedRunningTime="2026-11-02 21:59:19.447060975 +0000 UTC m=+3051.886635697" Nov 02 21:59:19.483360 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:19.483323 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" Nov 02 21:59:23.047223 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:23.047178 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:27.101232 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:27.101198 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:28.196960 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:28.196911 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:28.471861 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:59:28.471777 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:59:28.483017 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:28.482987 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" Nov 02 21:59:28.495620 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:28.495593 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" Nov 02 21:59:29.418208 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:29.418179 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" Nov 02 21:59:31.991359 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:31.991319 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl"] Nov 02 21:59:32.130256 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:32.130217 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:33.224682 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:33.224645 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:34.518736 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:34.518697 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:38.222395 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:38.222354 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:42.219920 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:42.219879 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:43.226716 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:43.226674 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:47.011851 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.011783 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" podUID="39599ffa-7e36-4a75-b0f7-efacc85bda8f" containerName="main" containerID="cri-o://cc21b4f63161de008178642f9f15d6f7918de1e625ea24abe00b515ae9f1953f" gracePeriod=45 Nov 02 21:59:47.302820 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.302795 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" Nov 02 21:59:47.396748 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.396710 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39599ffa-7e36-4a75-b0f7-efacc85bda8f-kube-api-access-92jt4\" (UniqueName: \"kubernetes.io/projected/39599ffa-7e36-4a75-b0f7-efacc85bda8f-kube-api-access-92jt4\") pod \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\" (UID: \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\") " Nov 02 21:59:47.396950 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.396784 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39599ffa-7e36-4a75-b0f7-efacc85bda8f-tls-certs\" (UniqueName: \"kubernetes.io/secret/39599ffa-7e36-4a75-b0f7-efacc85bda8f-tls-certs\") pod \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\" (UID: \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\") " Nov 02 21:59:47.396950 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.396856 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39599ffa-7e36-4a75-b0f7-efacc85bda8f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39599ffa-7e36-4a75-b0f7-efacc85bda8f-tmp-dir\") pod \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\" (UID: \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\") " Nov 02 21:59:47.396950 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.396930 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39599ffa-7e36-4a75-b0f7-efacc85bda8f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/39599ffa-7e36-4a75-b0f7-efacc85bda8f-dshm\") pod \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\" (UID: \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\") " Nov 02 21:59:47.397154 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.396993 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39599ffa-7e36-4a75-b0f7-efacc85bda8f-home\" (UniqueName: \"kubernetes.io/empty-dir/39599ffa-7e36-4a75-b0f7-efacc85bda8f-home\") pod \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\" (UID: \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\") " Nov 02 21:59:47.397154 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.397036 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39599ffa-7e36-4a75-b0f7-efacc85bda8f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39599ffa-7e36-4a75-b0f7-efacc85bda8f-model-cache\") pod \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\" (UID: \"39599ffa-7e36-4a75-b0f7-efacc85bda8f\") " Nov 02 21:59:47.397154 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.397093 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39599ffa-7e36-4a75-b0f7-efacc85bda8f-tmp-dir" pod "39599ffa-7e36-4a75-b0f7-efacc85bda8f" (UID: "39599ffa-7e36-4a75-b0f7-efacc85bda8f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:47.397334 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.397306 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39599ffa-7e36-4a75-b0f7-efacc85bda8f-home" pod "39599ffa-7e36-4a75-b0f7-efacc85bda8f" (UID: "39599ffa-7e36-4a75-b0f7-efacc85bda8f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:47.397414 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.397394 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39599ffa-7e36-4a75-b0f7-efacc85bda8f-model-cache" pod "39599ffa-7e36-4a75-b0f7-efacc85bda8f" (UID: "39599ffa-7e36-4a75-b0f7-efacc85bda8f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:47.397645 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.397625 4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39599ffa-7e36-4a75-b0f7-efacc85bda8f-tmp-dir\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:47.397730 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.397651 4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39599ffa-7e36-4a75-b0f7-efacc85bda8f-home\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:47.397730 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.397666 4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39599ffa-7e36-4a75-b0f7-efacc85bda8f-model-cache\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:47.399279 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.399246 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39599ffa-7e36-4a75-b0f7-efacc85bda8f-tls-certs" pod "39599ffa-7e36-4a75-b0f7-efacc85bda8f" (UID: "39599ffa-7e36-4a75-b0f7-efacc85bda8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:59:47.399485 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.399468 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39599ffa-7e36-4a75-b0f7-efacc85bda8f-dshm" pod "39599ffa-7e36-4a75-b0f7-efacc85bda8f" (UID: "39599ffa-7e36-4a75-b0f7-efacc85bda8f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:47.399565 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.399546 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39599ffa-7e36-4a75-b0f7-efacc85bda8f-kube-api-access-92jt4" pod "39599ffa-7e36-4a75-b0f7-efacc85bda8f" (UID: "39599ffa-7e36-4a75-b0f7-efacc85bda8f"). InnerVolumeSpecName "kube-api-access-92jt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:59:47.499202 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.499168 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92jt4\" (UniqueName: \"kubernetes.io/projected/39599ffa-7e36-4a75-b0f7-efacc85bda8f-kube-api-access-92jt4\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:47.499202 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.499200 4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39599ffa-7e36-4a75-b0f7-efacc85bda8f-tls-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:47.499202 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.499210 4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39599ffa-7e36-4a75-b0f7-efacc85bda8f-dshm\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:47.531303 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.531210 4935 generic.go:356] "Generic (PLEG): container finished" podID="39599ffa-7e36-4a75-b0f7-efacc85bda8f" containerID="cc21b4f63161de008178642f9f15d6f7918de1e625ea24abe00b515ae9f1953f" exitCode=0 Nov 02 21:59:47.531303 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.531265 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" Nov 02 21:59:47.531530 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.531304 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" event={"ID":"39599ffa-7e36-4a75-b0f7-efacc85bda8f","Type":"ContainerDied","Data":"cc21b4f63161de008178642f9f15d6f7918de1e625ea24abe00b515ae9f1953f"} Nov 02 21:59:47.531530 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.531352 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" event={"ID":"39599ffa-7e36-4a75-b0f7-efacc85bda8f","Type":"ContainerDied","Data":"287e51eb785983b891f3c49cbca851cf3786dddbf04f026a833471da47f48678"} Nov 02 21:59:47.531530 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.531375 4935 scope.go:122] "RemoveContainer" containerID="cc21b4f63161de008178642f9f15d6f7918de1e625ea24abe00b515ae9f1953f" Nov 02 21:59:47.542150 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.542107 4935 scope.go:122] "RemoveContainer" containerID="cc21b4f63161de008178642f9f15d6f7918de1e625ea24abe00b515ae9f1953f" Nov 02 21:59:47.542459 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:59:47.542442 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc21b4f63161de008178642f9f15d6f7918de1e625ea24abe00b515ae9f1953f\": container with ID starting with cc21b4f63161de008178642f9f15d6f7918de1e625ea24abe00b515ae9f1953f not found: ID does not exist" containerID="cc21b4f63161de008178642f9f15d6f7918de1e625ea24abe00b515ae9f1953f" Nov 02 21:59:47.542545 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.542474 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc21b4f63161de008178642f9f15d6f7918de1e625ea24abe00b515ae9f1953f"} err="failed to get container status \"cc21b4f63161de008178642f9f15d6f7918de1e625ea24abe00b515ae9f1953f\": rpc error: code = NotFound desc = could not find container \"cc21b4f63161de008178642f9f15d6f7918de1e625ea24abe00b515ae9f1953f\": container with ID starting with cc21b4f63161de008178642f9f15d6f7918de1e625ea24abe00b515ae9f1953f not found: ID does not exist" Nov 02 21:59:47.685313 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.685273 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl"] Nov 02 21:59:47.689442 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.689415 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl"] Nov 02 21:59:47.777805 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:47.777764 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39599ffa-7e36-4a75-b0f7-efacc85bda8f" path="/var/lib/kubelet/pods/39599ffa-7e36-4a75-b0f7-efacc85bda8f/volumes" Nov 02 21:59:48.192639 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:48.192597 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:48.194518 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:48.194481 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-explicit-4ca09f/explicit-model-4ca09f-kserve-6488fddb7c-mzrjl" podUID="39599ffa-7e36-4a75-b0f7-efacc85bda8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.126:8000/ready\": context deadline exceeded" Nov 02 21:59:49.288461 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.288426 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw"] Nov 02 21:59:49.288969 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.288956 4935 cpu_manager.go:414] "containerMap: removing container" podUID="39599ffa-7e36-4a75-b0f7-efacc85bda8f" containerName="main" Nov 02 21:59:49.289018 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.288970 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39599ffa-7e36-4a75-b0f7-efacc85bda8f" containerName="main" Nov 02 21:59:49.289101 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.289077 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="39599ffa-7e36-4a75-b0f7-efacc85bda8f" containerName="main" Nov 02 21:59:49.293785 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.293764 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" Nov 02 21:59:49.345388 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.345354 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw"] Nov 02 21:59:49.387483 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.387445 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-0f0874\"/\"openshift-service-ca.crt\"" Nov 02 21:59:49.387483 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.387480 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-0f0874\"/\"kube-root-ca.crt\"" Nov 02 21:59:49.389342 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.389325 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-0f0874\"/\"default-dockercfg-r6mm5\"" Nov 02 21:59:49.389646 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.389629 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-0f0874\"/\"auth-test-model-0f0874-kserve-self-signed-certs\"" Nov 02 21:59:49.422850 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.422816 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59e43da6-a639-4412-a666-93c5d8b04120-tls-certs\") pod \"auth-test-model-0f0874-kserve-79888dfff7-ggjhw\" (UID: \"59e43da6-a639-4412-a666-93c5d8b04120\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" Nov 02 21:59:49.423030 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.422943 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59e43da6-a639-4412-a666-93c5d8b04120-model-cache\") pod \"auth-test-model-0f0874-kserve-79888dfff7-ggjhw\" (UID: \"59e43da6-a639-4412-a666-93c5d8b04120\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" Nov 02 21:59:49.423030 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.422997 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59e43da6-a639-4412-a666-93c5d8b04120-tmp-dir\") pod \"auth-test-model-0f0874-kserve-79888dfff7-ggjhw\" (UID: \"59e43da6-a639-4412-a666-93c5d8b04120\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" Nov 02 21:59:49.423187 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.423031 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59e43da6-a639-4412-a666-93c5d8b04120-dshm\") pod \"auth-test-model-0f0874-kserve-79888dfff7-ggjhw\" (UID: \"59e43da6-a639-4412-a666-93c5d8b04120\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" Nov 02 21:59:49.423187 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.423064 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59e43da6-a639-4412-a666-93c5d8b04120-home\") pod \"auth-test-model-0f0874-kserve-79888dfff7-ggjhw\" (UID: \"59e43da6-a639-4412-a666-93c5d8b04120\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" Nov 02 21:59:49.423187 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.423167 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4spk\" (UniqueName: \"kubernetes.io/projected/59e43da6-a639-4412-a666-93c5d8b04120-kube-api-access-t4spk\") pod \"auth-test-model-0f0874-kserve-79888dfff7-ggjhw\" (UID: \"59e43da6-a639-4412-a666-93c5d8b04120\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" Nov 02 21:59:49.459289 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.459252 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:49.524169 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.524122 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4spk\" (UniqueName: \"kubernetes.io/projected/59e43da6-a639-4412-a666-93c5d8b04120-kube-api-access-t4spk\") pod \"auth-test-model-0f0874-kserve-79888dfff7-ggjhw\" (UID: \"59e43da6-a639-4412-a666-93c5d8b04120\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" Nov 02 21:59:49.524347 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.524229 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59e43da6-a639-4412-a666-93c5d8b04120-tls-certs\") pod \"auth-test-model-0f0874-kserve-79888dfff7-ggjhw\" (UID: \"59e43da6-a639-4412-a666-93c5d8b04120\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" Nov 02 21:59:49.524347 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.524329 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59e43da6-a639-4412-a666-93c5d8b04120-model-cache\") pod \"auth-test-model-0f0874-kserve-79888dfff7-ggjhw\" (UID: \"59e43da6-a639-4412-a666-93c5d8b04120\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" Nov 02 21:59:49.524477 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.524381 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59e43da6-a639-4412-a666-93c5d8b04120-tmp-dir\") pod \"auth-test-model-0f0874-kserve-79888dfff7-ggjhw\" (UID: \"59e43da6-a639-4412-a666-93c5d8b04120\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" Nov 02 21:59:49.524477 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.524415 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59e43da6-a639-4412-a666-93c5d8b04120-dshm\") pod \"auth-test-model-0f0874-kserve-79888dfff7-ggjhw\" (UID: \"59e43da6-a639-4412-a666-93c5d8b04120\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" Nov 02 21:59:49.524477 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.524444 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59e43da6-a639-4412-a666-93c5d8b04120-home\") pod \"auth-test-model-0f0874-kserve-79888dfff7-ggjhw\" (UID: \"59e43da6-a639-4412-a666-93c5d8b04120\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" Nov 02 21:59:49.524845 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.524812 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59e43da6-a639-4412-a666-93c5d8b04120-model-cache\") pod \"auth-test-model-0f0874-kserve-79888dfff7-ggjhw\" (UID: \"59e43da6-a639-4412-a666-93c5d8b04120\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" Nov 02 21:59:49.524992 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.524822 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59e43da6-a639-4412-a666-93c5d8b04120-tmp-dir\") pod \"auth-test-model-0f0874-kserve-79888dfff7-ggjhw\" (UID: \"59e43da6-a639-4412-a666-93c5d8b04120\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" Nov 02 21:59:49.524992 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.524894 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59e43da6-a639-4412-a666-93c5d8b04120-home\") pod \"auth-test-model-0f0874-kserve-79888dfff7-ggjhw\" (UID: \"59e43da6-a639-4412-a666-93c5d8b04120\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" Nov 02 21:59:49.526783 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.526755 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59e43da6-a639-4412-a666-93c5d8b04120-dshm\") pod \"auth-test-model-0f0874-kserve-79888dfff7-ggjhw\" (UID: \"59e43da6-a639-4412-a666-93c5d8b04120\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" Nov 02 21:59:49.527064 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.527043 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59e43da6-a639-4412-a666-93c5d8b04120-tls-certs\") pod \"auth-test-model-0f0874-kserve-79888dfff7-ggjhw\" (UID: \"59e43da6-a639-4412-a666-93c5d8b04120\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" Nov 02 21:59:49.577516 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.577435 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4spk\" (UniqueName: \"kubernetes.io/projected/59e43da6-a639-4412-a666-93c5d8b04120-kube-api-access-t4spk\") pod \"auth-test-model-0f0874-kserve-79888dfff7-ggjhw\" (UID: \"59e43da6-a639-4412-a666-93c5d8b04120\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" Nov 02 21:59:49.609693 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.609655 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" Nov 02 21:59:49.985721 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:49.985696 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw"] Nov 02 21:59:49.987396 ip-10-0-128-89 kubenswrapper[4935]: W1102 21:59:49.987367 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59e43da6_a639_4412_a666_93c5d8b04120.slice/crio-d425066f7835377fbd0e43e840b5cb3c9e2b7a1bc117e3ea4962983ea32fb77a WatchSource:0}: Error finding container d425066f7835377fbd0e43e840b5cb3c9e2b7a1bc117e3ea4962983ea32fb77a: Status 404 returned error can't find the container with id d425066f7835377fbd0e43e840b5cb3c9e2b7a1bc117e3ea4962983ea32fb77a Nov 02 21:59:50.556040 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:50.555998 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" event={"ID":"59e43da6-a639-4412-a666-93c5d8b04120","Type":"ContainerStarted","Data":"e7797e3b1b7100ceee1ea6e8a85b1378b405a159136cd4b15a785c6f0923f57a"} Nov 02 21:59:50.556457 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:50.556054 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" event={"ID":"59e43da6-a639-4412-a666-93c5d8b04120","Type":"ContainerStarted","Data":"d425066f7835377fbd0e43e840b5cb3c9e2b7a1bc117e3ea4962983ea32fb77a"} Nov 02 21:59:50.610819 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:50.610778 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" Nov 02 21:59:50.665846 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:50.665797 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" podStartSLOduration=1.6657836019999999 podStartE2EDuration="1.665783602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:59:50.662665182 +0000 UTC m=+3083.102239917" watchObservedRunningTime="2026-11-02 21:59:50.665783602 +0000 UTC m=+3083.105358323" Nov 02 21:59:53.217058 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:53.216992 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:58.207799 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:58.207764 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 21:59:58.469563 ip-10-0-128-89 kubenswrapper[4935]: E1102 21:59:58.469479 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 21:59:59.610687 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:59.610649 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" Nov 02 21:59:59.623796 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:59.623766 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" Nov 02 21:59:59.634883 ip-10-0-128-89 kubenswrapper[4935]: I1102 21:59:59.634853 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" Nov 02 22:00:00.214984 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:00.214949 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm"] Nov 02 22:00:00.218920 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:00.218893 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm" Nov 02 22:00:00.220974 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:00.220946 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2"] Nov 02 22:00:00.224565 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:00.224542 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2" Nov 02 22:00:00.256994 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:00.256952 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6z4m\" (UniqueName: \"kubernetes.io/projected/9cb8d510-bb7b-4598-886b-c26a6f891576-kube-api-access-n6z4m\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2\" (UID: \"9cb8d510-bb7b-4598-886b-c26a6f891576\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2" Nov 02 22:00:00.257218 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:00.257195 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cb8d510-bb7b-4598-886b-c26a6f891576-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2\" (UID: \"9cb8d510-bb7b-4598-886b-c26a6f891576\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2" Nov 02 22:00:00.257322 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:00.257231 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7c556b2-03fd-4529-9046-a385270e2d13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm\" (UID: \"f7c556b2-03fd-4529-9046-a385270e2d13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm" Nov 02 22:00:00.257322 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:00.257258 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp287\" (UniqueName: \"kubernetes.io/projected/f7c556b2-03fd-4529-9046-a385270e2d13-kube-api-access-tp287\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm\" (UID: \"f7c556b2-03fd-4529-9046-a385270e2d13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm" Nov 02 22:00:00.323155 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:00.314118 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm"] Nov 02 22:00:00.323155 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:00.318585 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2"] Nov 02 22:00:00.355597 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:00.355571 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\"" Nov 02 22:00:00.358234 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:00.358188 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cb8d510-bb7b-4598-886b-c26a6f891576-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2\" (UID: \"9cb8d510-bb7b-4598-886b-c26a6f891576\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2" Nov 02 22:00:00.358391 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:00.358243 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7c556b2-03fd-4529-9046-a385270e2d13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm\" (UID: \"f7c556b2-03fd-4529-9046-a385270e2d13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm" Nov 02 22:00:00.358391 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:00.358275 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp287\" (UniqueName: \"kubernetes.io/projected/f7c556b2-03fd-4529-9046-a385270e2d13-kube-api-access-tp287\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm\" (UID: \"f7c556b2-03fd-4529-9046-a385270e2d13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm" Nov 02 22:00:00.358391 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:00.358376 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6z4m\" (UniqueName: \"kubernetes.io/projected/9cb8d510-bb7b-4598-886b-c26a6f891576-kube-api-access-n6z4m\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2\" (UID: \"9cb8d510-bb7b-4598-886b-c26a6f891576\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2" Nov 02 22:00:00.359068 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:00.359042 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cb8d510-bb7b-4598-886b-c26a6f891576-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2\" (UID: \"9cb8d510-bb7b-4598-886b-c26a6f891576\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2" Nov 02 22:00:00.359273 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:00.359256 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7c556b2-03fd-4529-9046-a385270e2d13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm\" (UID: \"f7c556b2-03fd-4529-9046-a385270e2d13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm" Nov 02 22:00:00.412309 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:00.412267 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp287\" (UniqueName: \"kubernetes.io/projected/f7c556b2-03fd-4529-9046-a385270e2d13-kube-api-access-tp287\") pod \"maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm\" (UID: \"f7c556b2-03fd-4529-9046-a385270e2d13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm" Nov 02 22:00:00.412467 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:00.412311 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6z4m\" (UniqueName: \"kubernetes.io/projected/9cb8d510-bb7b-4598-886b-c26a6f891576-kube-api-access-n6z4m\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2\" (UID: \"9cb8d510-bb7b-4598-886b-c26a6f891576\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2" Nov 02 22:00:00.536355 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:00.536323 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm" Nov 02 22:00:00.541614 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:00.541583 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2" Nov 02 22:00:00.918395 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:00.918358 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm"] Nov 02 22:00:00.919387 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:00:00.919355 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c556b2_03fd_4529_9046_a385270e2d13.slice/crio-c18f0d567437ca057296c40aadfea427e0351e86248534385c4da1ffcb8b4cd6 WatchSource:0}: Error finding container c18f0d567437ca057296c40aadfea427e0351e86248534385c4da1ffcb8b4cd6: Status 404 returned error can't find the container with id c18f0d567437ca057296c40aadfea427e0351e86248534385c4da1ffcb8b4cd6 Nov 02 22:00:00.966951 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:00:00.966919 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb8d510_bb7b_4598_886b_c26a6f891576.slice/crio-77a32716f85f127a69b83e223532cdba25aa73bd960eb3b202f3523848ebc426 WatchSource:0}: Error finding container 77a32716f85f127a69b83e223532cdba25aa73bd960eb3b202f3523848ebc426: Status 404 returned error can't find the container with id 77a32716f85f127a69b83e223532cdba25aa73bd960eb3b202f3523848ebc426 Nov 02 22:00:00.967274 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:00.967244 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2"] Nov 02 22:00:01.643197 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:01.643160 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm" event={"ID":"f7c556b2-03fd-4529-9046-a385270e2d13","Type":"ContainerStarted","Data":"de8ec426053c132af8fdc96e9d2cc837f7f215ca724574200aefacd9e1ba0cfc"} Nov 02 22:00:01.643400 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:01.643216 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm" event={"ID":"f7c556b2-03fd-4529-9046-a385270e2d13","Type":"ContainerStarted","Data":"c18f0d567437ca057296c40aadfea427e0351e86248534385c4da1ffcb8b4cd6"} Nov 02 22:00:01.644836 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:01.644811 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2" event={"ID":"9cb8d510-bb7b-4598-886b-c26a6f891576","Type":"ContainerStarted","Data":"e1a4925c9bc371546982fda73da0102ce05dc3a9399cce0a3a66ed0faa396457"} Nov 02 22:00:01.644940 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:01.644855 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2" event={"ID":"9cb8d510-bb7b-4598-886b-c26a6f891576","Type":"ContainerStarted","Data":"77a32716f85f127a69b83e223532cdba25aa73bd960eb3b202f3523848ebc426"} Nov 02 22:00:01.749675 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:01.749620 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm" podStartSLOduration=1.749605341 podStartE2EDuration="1.749605341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:01.748347056 +0000 UTC m=+3094.187921780" watchObservedRunningTime="2026-11-02 22:00:01.749605341 +0000 UTC m=+3094.189180063" Nov 02 22:00:01.856144 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:01.856090 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2" podStartSLOduration=1.8560773990000001 podStartE2EDuration="1.856077399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:00:01.853519572 +0000 UTC m=+3094.293094298" watchObservedRunningTime="2026-11-02 22:00:01.856077399 +0000 UTC m=+3094.295652121" Nov 02 22:00:02.653299 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:02.653263 4935 generic.go:356] "Generic (PLEG): container finished" podID="9cb8d510-bb7b-4598-886b-c26a6f891576" containerID="e1a4925c9bc371546982fda73da0102ce05dc3a9399cce0a3a66ed0faa396457" exitCode=60 Nov 02 22:00:02.653801 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:02.653351 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2" event={"ID":"9cb8d510-bb7b-4598-886b-c26a6f891576","Type":"ContainerDied","Data":"e1a4925c9bc371546982fda73da0102ce05dc3a9399cce0a3a66ed0faa396457"} Nov 02 22:00:02.653801 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:02.653615 4935 scope.go:122] "RemoveContainer" containerID="e1a4925c9bc371546982fda73da0102ce05dc3a9399cce0a3a66ed0faa396457" Nov 02 22:00:02.655729 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:02.655609 4935 generic.go:356] "Generic (PLEG): container finished" podID="f7c556b2-03fd-4529-9046-a385270e2d13" containerID="de8ec426053c132af8fdc96e9d2cc837f7f215ca724574200aefacd9e1ba0cfc" exitCode=60 Nov 02 22:00:02.655729 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:02.655667 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm" event={"ID":"f7c556b2-03fd-4529-9046-a385270e2d13","Type":"ContainerDied","Data":"de8ec426053c132af8fdc96e9d2cc837f7f215ca724574200aefacd9e1ba0cfc"} Nov 02 22:00:02.656006 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:02.655986 4935 scope.go:122] "RemoveContainer" containerID="de8ec426053c132af8fdc96e9d2cc837f7f215ca724574200aefacd9e1ba0cfc" Nov 02 22:00:03.245538 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:03.245506 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:03.665729 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:03.665690 4935 generic.go:356] "Generic (PLEG): container finished" podID="f7c556b2-03fd-4529-9046-a385270e2d13" containerID="48017229d37b58a69d65bec642d17362de5cb1d53da2cbaa30aebb8b1cac83b3" exitCode=60 Nov 02 22:00:03.666304 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:03.665873 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm" event={"ID":"f7c556b2-03fd-4529-9046-a385270e2d13","Type":"ContainerDied","Data":"48017229d37b58a69d65bec642d17362de5cb1d53da2cbaa30aebb8b1cac83b3"} Nov 02 22:00:03.666304 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:03.665944 4935 scope.go:122] "RemoveContainer" containerID="de8ec426053c132af8fdc96e9d2cc837f7f215ca724574200aefacd9e1ba0cfc" Nov 02 22:00:03.666304 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:03.666157 4935 scope.go:122] "RemoveContainer" containerID="48017229d37b58a69d65bec642d17362de5cb1d53da2cbaa30aebb8b1cac83b3" Nov 02 22:00:03.666512 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:00:03.666487 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm_odh-ai-gateway-infra(f7c556b2-03fd-4529-9046-a385270e2d13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm" podUID="f7c556b2-03fd-4529-9046-a385270e2d13" Nov 02 22:00:03.669380 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:03.669356 4935 generic.go:356] "Generic (PLEG): container finished" podID="9cb8d510-bb7b-4598-886b-c26a6f891576" containerID="ed22bf0d50c73805b03ca0ad6fc780d6f0b09d19c635f880ba965ced729997f9" exitCode=60 Nov 02 22:00:03.669502 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:03.669404 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2" event={"ID":"9cb8d510-bb7b-4598-886b-c26a6f891576","Type":"ContainerDied","Data":"ed22bf0d50c73805b03ca0ad6fc780d6f0b09d19c635f880ba965ced729997f9"} Nov 02 22:00:03.669708 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:03.669692 4935 scope.go:122] "RemoveContainer" containerID="ed22bf0d50c73805b03ca0ad6fc780d6f0b09d19c635f880ba965ced729997f9" Nov 02 22:00:03.670005 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:00:03.669984 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2_odh-ai-gateway-infra(9cb8d510-bb7b-4598-886b-c26a6f891576)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2" podUID="9cb8d510-bb7b-4598-886b-c26a6f891576" Nov 02 22:00:03.680389 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:03.680371 4935 scope.go:122] "RemoveContainer" containerID="e1a4925c9bc371546982fda73da0102ce05dc3a9399cce0a3a66ed0faa396457" Nov 02 22:00:04.679044 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:04.679015 4935 scope.go:122] "RemoveContainer" containerID="48017229d37b58a69d65bec642d17362de5cb1d53da2cbaa30aebb8b1cac83b3" Nov 02 22:00:04.679515 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:00:04.679331 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm_odh-ai-gateway-infra(f7c556b2-03fd-4529-9046-a385270e2d13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm" podUID="f7c556b2-03fd-4529-9046-a385270e2d13" Nov 02 22:00:04.680647 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:04.680627 4935 scope.go:122] "RemoveContainer" containerID="ed22bf0d50c73805b03ca0ad6fc780d6f0b09d19c635f880ba965ced729997f9" Nov 02 22:00:04.680876 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:00:04.680859 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2_odh-ai-gateway-infra(9cb8d510-bb7b-4598-886b-c26a6f891576)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2" podUID="9cb8d510-bb7b-4598-886b-c26a6f891576" Nov 02 22:00:08.211586 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:08.211489 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:08.995704 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:08.995669 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:10.426935 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:10.426890 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:10.630226 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:10.630188 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:13.664737 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:13.664701 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:14.560297 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:14.560253 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:14.769974 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:14.769946 4935 scope.go:122] "RemoveContainer" containerID="ed22bf0d50c73805b03ca0ad6fc780d6f0b09d19c635f880ba965ced729997f9" Nov 02 22:00:15.752361 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:15.752324 4935 generic.go:356] "Generic (PLEG): container finished" podID="9cb8d510-bb7b-4598-886b-c26a6f891576" containerID="18a2954a2956dc4ba57efad1ab6be9b446af51a8567f98dcf2fb60ef294f3f0b" exitCode=60 Nov 02 22:00:15.752529 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:15.752394 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2" event={"ID":"9cb8d510-bb7b-4598-886b-c26a6f891576","Type":"ContainerDied","Data":"18a2954a2956dc4ba57efad1ab6be9b446af51a8567f98dcf2fb60ef294f3f0b"} Nov 02 22:00:15.752529 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:15.752456 4935 scope.go:122] "RemoveContainer" containerID="ed22bf0d50c73805b03ca0ad6fc780d6f0b09d19c635f880ba965ced729997f9" Nov 02 22:00:15.752860 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:15.752843 4935 scope.go:122] "RemoveContainer" containerID="18a2954a2956dc4ba57efad1ab6be9b446af51a8567f98dcf2fb60ef294f3f0b" Nov 02 22:00:15.753173 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:00:15.753151 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2_odh-ai-gateway-infra(9cb8d510-bb7b-4598-886b-c26a6f891576)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2" podUID="9cb8d510-bb7b-4598-886b-c26a6f891576" Nov 02 22:00:15.769488 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:15.769473 4935 scope.go:122] "RemoveContainer" containerID="48017229d37b58a69d65bec642d17362de5cb1d53da2cbaa30aebb8b1cac83b3" Nov 02 22:00:16.763197 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:16.763153 4935 generic.go:356] "Generic (PLEG): container finished" podID="f7c556b2-03fd-4529-9046-a385270e2d13" containerID="fbd5c64452fe6eded0d23b18f8fa2210451da90229f9dd50f68c4e7c4dd6ad64" exitCode=60 Nov 02 22:00:16.763689 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:16.763314 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm" event={"ID":"f7c556b2-03fd-4529-9046-a385270e2d13","Type":"ContainerDied","Data":"fbd5c64452fe6eded0d23b18f8fa2210451da90229f9dd50f68c4e7c4dd6ad64"} Nov 02 22:00:16.763689 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:16.763409 4935 scope.go:122] "RemoveContainer" containerID="48017229d37b58a69d65bec642d17362de5cb1d53da2cbaa30aebb8b1cac83b3" Nov 02 22:00:16.763998 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:16.763972 4935 scope.go:122] "RemoveContainer" containerID="fbd5c64452fe6eded0d23b18f8fa2210451da90229f9dd50f68c4e7c4dd6ad64" Nov 02 22:00:16.768894 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:00:16.768858 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm_odh-ai-gateway-infra(f7c556b2-03fd-4529-9046-a385270e2d13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm" podUID="f7c556b2-03fd-4529-9046-a385270e2d13" Nov 02 22:00:16.869776 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:16.869744 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2"] Nov 02 22:00:17.023105 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:17.023081 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2" Nov 02 22:00:17.162349 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:17.162315 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb8d510-bb7b-4598-886b-c26a6f891576-kube-api-access-n6z4m\" (UniqueName: \"kubernetes.io/projected/9cb8d510-bb7b-4598-886b-c26a6f891576-kube-api-access-n6z4m\") pod \"9cb8d510-bb7b-4598-886b-c26a6f891576\" (UID: \"9cb8d510-bb7b-4598-886b-c26a6f891576\") " Nov 02 22:00:17.162555 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:17.162369 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cb8d510-bb7b-4598-886b-c26a6f891576-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cb8d510-bb7b-4598-886b-c26a6f891576-maas-api-service-ca\") pod \"9cb8d510-bb7b-4598-886b-c26a6f891576\" (UID: \"9cb8d510-bb7b-4598-886b-c26a6f891576\") " Nov 02 22:00:17.162794 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:17.162762 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cb8d510-bb7b-4598-886b-c26a6f891576-maas-api-service-ca" pod "9cb8d510-bb7b-4598-886b-c26a6f891576" (UID: "9cb8d510-bb7b-4598-886b-c26a6f891576"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:17.162964 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:17.162949 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cb8d510-bb7b-4598-886b-c26a6f891576-maas-api-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:17.164526 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:17.164501 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb8d510-bb7b-4598-886b-c26a6f891576-kube-api-access-n6z4m" pod "9cb8d510-bb7b-4598-886b-c26a6f891576" (UID: "9cb8d510-bb7b-4598-886b-c26a6f891576"). InnerVolumeSpecName "kube-api-access-n6z4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:17.263978 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:17.263946 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6z4m\" (UniqueName: \"kubernetes.io/projected/9cb8d510-bb7b-4598-886b-c26a6f891576-kube-api-access-n6z4m\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:17.775933 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:17.775899 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2" Nov 02 22:00:17.778534 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:17.778504 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2" event={"ID":"9cb8d510-bb7b-4598-886b-c26a6f891576","Type":"ContainerDied","Data":"77a32716f85f127a69b83e223532cdba25aa73bd960eb3b202f3523848ebc426"} Nov 02 22:00:17.778674 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:17.778561 4935 scope.go:122] "RemoveContainer" containerID="18a2954a2956dc4ba57efad1ab6be9b446af51a8567f98dcf2fb60ef294f3f0b" Nov 02 22:00:17.883017 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:17.882981 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm"] Nov 02 22:00:17.937707 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:17.932160 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2"] Nov 02 22:00:17.937707 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:17.936765 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0f0874-29894280-zxcq2"] Nov 02 22:00:18.044925 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:18.044864 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm" Nov 02 22:00:18.176121 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:18.176089 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7c556b2-03fd-4529-9046-a385270e2d13-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7c556b2-03fd-4529-9046-a385270e2d13-maas-api-service-ca\") pod \"f7c556b2-03fd-4529-9046-a385270e2d13\" (UID: \"f7c556b2-03fd-4529-9046-a385270e2d13\") " Nov 02 22:00:18.176323 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:18.176149 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7c556b2-03fd-4529-9046-a385270e2d13-kube-api-access-tp287\" (UniqueName: \"kubernetes.io/projected/f7c556b2-03fd-4529-9046-a385270e2d13-kube-api-access-tp287\") pod \"f7c556b2-03fd-4529-9046-a385270e2d13\" (UID: \"f7c556b2-03fd-4529-9046-a385270e2d13\") " Nov 02 22:00:18.176582 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:18.176554 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7c556b2-03fd-4529-9046-a385270e2d13-maas-api-service-ca" pod "f7c556b2-03fd-4529-9046-a385270e2d13" (UID: "f7c556b2-03fd-4529-9046-a385270e2d13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:18.178323 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:18.178297 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7c556b2-03fd-4529-9046-a385270e2d13-kube-api-access-tp287" pod "f7c556b2-03fd-4529-9046-a385270e2d13" (UID: "f7c556b2-03fd-4529-9046-a385270e2d13"). InnerVolumeSpecName "kube-api-access-tp287". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:18.277792 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:18.277755 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp287\" (UniqueName: \"kubernetes.io/projected/f7c556b2-03fd-4529-9046-a385270e2d13-kube-api-access-tp287\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:18.277792 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:18.277785 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7c556b2-03fd-4529-9046-a385270e2d13-maas-api-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:18.787064 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:18.787024 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm" event={"ID":"f7c556b2-03fd-4529-9046-a385270e2d13","Type":"ContainerDied","Data":"c18f0d567437ca057296c40aadfea427e0351e86248534385c4da1ffcb8b4cd6"} Nov 02 22:00:18.787628 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:18.787090 4935 scope.go:122] "RemoveContainer" containerID="fbd5c64452fe6eded0d23b18f8fa2210451da90229f9dd50f68c4e7c4dd6ad64" Nov 02 22:00:18.787628 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:18.787039 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm" Nov 02 22:00:18.842906 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:18.842873 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:18.944403 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:18.944362 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm"] Nov 02 22:00:18.957069 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:18.957037 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-0c9014-29894280-677tm"] Nov 02 22:00:19.776908 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:19.776828 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cb8d510-bb7b-4598-886b-c26a6f891576" path="/var/lib/kubelet/pods/9cb8d510-bb7b-4598-886b-c26a6f891576/volumes" Nov 02 22:00:19.777299 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:19.777285 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7c556b2-03fd-4529-9046-a385270e2d13" path="/var/lib/kubelet/pods/f7c556b2-03fd-4529-9046-a385270e2d13/volumes" Nov 02 22:00:23.054280 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:23.054245 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj"] Nov 02 22:00:23.054832 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:23.054496 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj" podUID="297af528-36c0-423d-9bb4-0cbe3bec1bd3" containerName="maas-api" containerID="cri-o://31089b5cbf782ff73b6e45cac548037cec16e21a98babc000cf139ba6af2ad9f" gracePeriod=30 Nov 02 22:00:23.253639 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:23.253046 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:23.376358 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:23.376325 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj" Nov 02 22:00:23.450923 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:23.450458 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/297af528-36c0-423d-9bb4-0cbe3bec1bd3-kube-api-access-hncp7\" (UniqueName: \"kubernetes.io/projected/297af528-36c0-423d-9bb4-0cbe3bec1bd3-kube-api-access-hncp7\") pod \"297af528-36c0-423d-9bb4-0cbe3bec1bd3\" (UID: \"297af528-36c0-423d-9bb4-0cbe3bec1bd3\") " Nov 02 22:00:23.451113 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:23.450974 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/297af528-36c0-423d-9bb4-0cbe3bec1bd3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/297af528-36c0-423d-9bb4-0cbe3bec1bd3-openshift-service-ca\") pod \"297af528-36c0-423d-9bb4-0cbe3bec1bd3\" (UID: \"297af528-36c0-423d-9bb4-0cbe3bec1bd3\") " Nov 02 22:00:23.451374 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:23.451344 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/297af528-36c0-423d-9bb4-0cbe3bec1bd3-openshift-service-ca" pod "297af528-36c0-423d-9bb4-0cbe3bec1bd3" (UID: "297af528-36c0-423d-9bb4-0cbe3bec1bd3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:23.451487 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:23.451471 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/297af528-36c0-423d-9bb4-0cbe3bec1bd3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/297af528-36c0-423d-9bb4-0cbe3bec1bd3-maas-api-tls\") pod \"297af528-36c0-423d-9bb4-0cbe3bec1bd3\" (UID: \"297af528-36c0-423d-9bb4-0cbe3bec1bd3\") " Nov 02 22:00:23.451725 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:23.451702 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/297af528-36c0-423d-9bb4-0cbe3bec1bd3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/297af528-36c0-423d-9bb4-0cbe3bec1bd3-metrics-certs\") pod \"297af528-36c0-423d-9bb4-0cbe3bec1bd3\" (UID: \"297af528-36c0-423d-9bb4-0cbe3bec1bd3\") " Nov 02 22:00:23.452143 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:23.452110 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/297af528-36c0-423d-9bb4-0cbe3bec1bd3-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:23.469122 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:23.469085 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/297af528-36c0-423d-9bb4-0cbe3bec1bd3-kube-api-access-hncp7" pod "297af528-36c0-423d-9bb4-0cbe3bec1bd3" (UID: "297af528-36c0-423d-9bb4-0cbe3bec1bd3"). InnerVolumeSpecName "kube-api-access-hncp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:23.474291 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:23.474246 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/297af528-36c0-423d-9bb4-0cbe3bec1bd3-maas-api-tls" pod "297af528-36c0-423d-9bb4-0cbe3bec1bd3" (UID: "297af528-36c0-423d-9bb4-0cbe3bec1bd3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:00:23.474444 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:23.474335 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/297af528-36c0-423d-9bb4-0cbe3bec1bd3-metrics-certs" pod "297af528-36c0-423d-9bb4-0cbe3bec1bd3" (UID: "297af528-36c0-423d-9bb4-0cbe3bec1bd3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:00:23.553194 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:23.553120 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/297af528-36c0-423d-9bb4-0cbe3bec1bd3-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:23.553194 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:23.553183 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hncp7\" (UniqueName: \"kubernetes.io/projected/297af528-36c0-423d-9bb4-0cbe3bec1bd3-kube-api-access-hncp7\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:23.553194 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:23.553193 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/297af528-36c0-423d-9bb4-0cbe3bec1bd3-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:23.819695 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:23.819655 4935 generic.go:356] "Generic (PLEG): container finished" podID="297af528-36c0-423d-9bb4-0cbe3bec1bd3" containerID="31089b5cbf782ff73b6e45cac548037cec16e21a98babc000cf139ba6af2ad9f" exitCode=0 Nov 02 22:00:23.819836 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:23.819697 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj" Nov 02 22:00:23.819836 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:23.819693 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj" event={"ID":"297af528-36c0-423d-9bb4-0cbe3bec1bd3","Type":"ContainerDied","Data":"31089b5cbf782ff73b6e45cac548037cec16e21a98babc000cf139ba6af2ad9f"} Nov 02 22:00:23.819836 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:23.819814 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj" event={"ID":"297af528-36c0-423d-9bb4-0cbe3bec1bd3","Type":"ContainerDied","Data":"98cb39556d348cfedeeb19d6306024de15aefc093b7801fe53357a1ee65724e3"} Nov 02 22:00:23.819836 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:23.819836 4935 scope.go:122] "RemoveContainer" containerID="31089b5cbf782ff73b6e45cac548037cec16e21a98babc000cf139ba6af2ad9f" Nov 02 22:00:23.829917 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:23.829886 4935 scope.go:122] "RemoveContainer" containerID="31089b5cbf782ff73b6e45cac548037cec16e21a98babc000cf139ba6af2ad9f" Nov 02 22:00:23.830199 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:00:23.830177 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31089b5cbf782ff73b6e45cac548037cec16e21a98babc000cf139ba6af2ad9f\": container with ID starting with 31089b5cbf782ff73b6e45cac548037cec16e21a98babc000cf139ba6af2ad9f not found: ID does not exist" containerID="31089b5cbf782ff73b6e45cac548037cec16e21a98babc000cf139ba6af2ad9f" Nov 02 22:00:23.830271 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:23.830208 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31089b5cbf782ff73b6e45cac548037cec16e21a98babc000cf139ba6af2ad9f"} err="failed to get container status \"31089b5cbf782ff73b6e45cac548037cec16e21a98babc000cf139ba6af2ad9f\": rpc error: code = NotFound desc = could not find container \"31089b5cbf782ff73b6e45cac548037cec16e21a98babc000cf139ba6af2ad9f\": container with ID starting with 31089b5cbf782ff73b6e45cac548037cec16e21a98babc000cf139ba6af2ad9f not found: ID does not exist" Nov 02 22:00:23.973099 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:23.973062 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj"] Nov 02 22:00:23.976073 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:23.976052 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4ca09f-7b69b6865b-qhjhj"] Nov 02 22:00:25.037627 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:25.037574 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:25.776619 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:25.776572 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="297af528-36c0-423d-9bb4-0cbe3bec1bd3" path="/var/lib/kubelet/pods/297af528-36c0-423d-9bb4-0cbe3bec1bd3/volumes" Nov 02 22:00:28.469229 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:00:28.469193 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:00:28.832271 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:28.832233 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:29.034519 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:29.034484 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:30.177207 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:30.177170 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:30.375966 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:30.375931 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:30.964383 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:30.964339 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:31.603465 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:31.603432 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:32.215011 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:32.214976 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:32.604409 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:32.604373 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:33.993049 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:33.993011 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:34.203648 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:34.203616 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:34.434923 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:34.434877 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:34.872612 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:34.872573 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:35.731845 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:35.731795 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:40.745053 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:40.745006 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:41.128310 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:41.128266 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:46.182048 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:46.182011 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:51.129607 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:51.129565 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:56.113882 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:56.113834 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:58.003824 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:00:58.003782 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:00:58.472991 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:00:58.472951 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:01:01.128937 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:01.128889 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:05.007480 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:05.007440 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q"] Nov 02 22:01:05.008214 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:05.007702 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" podUID="83930246-18af-44a1-b4c6-3353bbba6814" containerName="istio-proxy" containerID="cri-o://2c2e7b8357b6413a93294126a40ad5f6d80736f8be5fa69c8deaffcd6a9298b6" gracePeriod=30 Nov 02 22:01:05.799660 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:05.799611 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:06.108320 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:06.107993 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:09.765816 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:09.765782 4935 patch_prober.go:29] interesting pod/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:01:09.766305 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:09.765840 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" podUID="83930246-18af-44a1-b4c6-3353bbba6814" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:01:10.158498 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.158457 4935 generic.go:356] "Generic (PLEG): container finished" podID="83930246-18af-44a1-b4c6-3353bbba6814" containerID="2c2e7b8357b6413a93294126a40ad5f6d80736f8be5fa69c8deaffcd6a9298b6" exitCode=0 Nov 02 22:01:10.158695 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.158524 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" event={"ID":"83930246-18af-44a1-b4c6-3353bbba6814","Type":"ContainerDied","Data":"2c2e7b8357b6413a93294126a40ad5f6d80736f8be5fa69c8deaffcd6a9298b6"} Nov 02 22:01:10.269740 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.269714 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 22:01:10.411958 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.411862 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83930246-18af-44a1-b4c6-3353bbba6814-kube-api-access-5kz7s\" (UniqueName: \"kubernetes.io/projected/83930246-18af-44a1-b4c6-3353bbba6814-kube-api-access-5kz7s\") pod \"83930246-18af-44a1-b4c6-3353bbba6814\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " Nov 02 22:01:10.411958 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.411930 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-workload-socket\") pod \"83930246-18af-44a1-b4c6-3353bbba6814\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " Nov 02 22:01:10.412238 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.412016 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-istio-envoy\") pod \"83930246-18af-44a1-b4c6-3353bbba6814\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " Nov 02 22:01:10.412238 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.412077 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83930246-18af-44a1-b4c6-3353bbba6814-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83930246-18af-44a1-b4c6-3353bbba6814-istio-podinfo\") pod \"83930246-18af-44a1-b4c6-3353bbba6814\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " Nov 02 22:01:10.412238 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.412117 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83930246-18af-44a1-b4c6-3353bbba6814-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83930246-18af-44a1-b4c6-3353bbba6814-istiod-ca-cert\") pod \"83930246-18af-44a1-b4c6-3353bbba6814\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " Nov 02 22:01:10.412238 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.412173 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-workload-certs\") pod \"83930246-18af-44a1-b4c6-3353bbba6814\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " Nov 02 22:01:10.412238 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.412215 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83930246-18af-44a1-b4c6-3353bbba6814-istio-token\" (UniqueName: \"kubernetes.io/projected/83930246-18af-44a1-b4c6-3353bbba6814-istio-token\") pod \"83930246-18af-44a1-b4c6-3353bbba6814\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " Nov 02 22:01:10.412523 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.412265 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-workload-socket" pod "83930246-18af-44a1-b4c6-3353bbba6814" (UID: "83930246-18af-44a1-b4c6-3353bbba6814"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:10.412523 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.412296 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-istio-data\") pod \"83930246-18af-44a1-b4c6-3353bbba6814\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " Nov 02 22:01:10.412523 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.412335 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-credential-socket\") pod \"83930246-18af-44a1-b4c6-3353bbba6814\" (UID: \"83930246-18af-44a1-b4c6-3353bbba6814\") " Nov 02 22:01:10.412523 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.412484 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-workload-certs" pod "83930246-18af-44a1-b4c6-3353bbba6814" (UID: "83930246-18af-44a1-b4c6-3353bbba6814"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:10.412745 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.412709 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-credential-socket" pod "83930246-18af-44a1-b4c6-3353bbba6814" (UID: "83930246-18af-44a1-b4c6-3353bbba6814"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:10.412848 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.412813 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83930246-18af-44a1-b4c6-3353bbba6814-istiod-ca-cert" pod "83930246-18af-44a1-b4c6-3353bbba6814" (UID: "83930246-18af-44a1-b4c6-3353bbba6814"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:01:10.412917 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.412850 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:10.412917 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.412865 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-istio-data" pod "83930246-18af-44a1-b4c6-3353bbba6814" (UID: "83930246-18af-44a1-b4c6-3353bbba6814"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:10.412917 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.412874 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:10.412917 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.412888 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:10.415047 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.415023 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83930246-18af-44a1-b4c6-3353bbba6814-istio-podinfo" pod "83930246-18af-44a1-b4c6-3353bbba6814" (UID: "83930246-18af-44a1-b4c6-3353bbba6814"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:01:10.415220 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.415094 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-istio-envoy" pod "83930246-18af-44a1-b4c6-3353bbba6814" (UID: "83930246-18af-44a1-b4c6-3353bbba6814"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:01:10.415220 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.415102 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83930246-18af-44a1-b4c6-3353bbba6814-kube-api-access-5kz7s" pod "83930246-18af-44a1-b4c6-3353bbba6814" (UID: "83930246-18af-44a1-b4c6-3353bbba6814"). InnerVolumeSpecName "kube-api-access-5kz7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:10.415331 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.415316 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83930246-18af-44a1-b4c6-3353bbba6814-istio-token" pod "83930246-18af-44a1-b4c6-3353bbba6814" (UID: "83930246-18af-44a1-b4c6-3353bbba6814"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:10.513605 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.513567 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83930246-18af-44a1-b4c6-3353bbba6814-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:10.513605 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.513597 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83930246-18af-44a1-b4c6-3353bbba6814-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:10.513605 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.513606 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83930246-18af-44a1-b4c6-3353bbba6814-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:10.513605 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.513615 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:10.513965 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.513624 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kz7s\" (UniqueName: \"kubernetes.io/projected/83930246-18af-44a1-b4c6-3353bbba6814-kube-api-access-5kz7s\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:10.513965 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:10.513633 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83930246-18af-44a1-b4c6-3353bbba6814-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:11.120030 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:11.119991 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:11.166076 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:11.166044 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" Nov 02 22:01:11.166263 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:11.166069 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q" event={"ID":"83930246-18af-44a1-b4c6-3353bbba6814","Type":"ContainerDied","Data":"e77ab65bd584eb3015cdb53bc19842ff2acdd72fa1e49c7ec407f638445fba9d"} Nov 02 22:01:11.166263 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:11.166147 4935 scope.go:122] "RemoveContainer" containerID="2c2e7b8357b6413a93294126a40ad5f6d80736f8be5fa69c8deaffcd6a9298b6" Nov 02 22:01:11.320178 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:11.320120 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q"] Nov 02 22:01:11.366791 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:11.366758 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-vsr7q"] Nov 02 22:01:11.778240 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:11.778198 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83930246-18af-44a1-b4c6-3353bbba6814" path="/var/lib/kubelet/pods/83930246-18af-44a1-b4c6-3353bbba6814/volumes" Nov 02 22:01:16.164553 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:16.164508 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:21.253303 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:21.253258 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:26.246548 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:26.246506 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:28.474716 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:01:28.474669 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:01:31.240004 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:31.239963 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:35.178253 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:35.178211 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:35.396386 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:35.396347 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:36.239776 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:36.239738 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:41.238162 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:41.238034 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:46.237277 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:46.237240 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:47.204574 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:47.204536 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:47.535345 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:47.535313 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:47.758911 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:47.758870 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:49.619732 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:49.619676 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:51.343397 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:51.343362 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:52.740721 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:52.740675 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:56.334683 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:01:56.334642 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:01:58.470608 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:01:58.470565 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:02:01.326863 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:02:01.326785 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:03.777312 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:02:03.777267 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:06.528107 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:02:06.528072 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:13.770904 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:02:13.770860 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:16.651017 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:02:16.650972 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:21.720101 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:02:21.720065 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:26.736558 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:02:26.736523 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:28.470256 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:02:28.470221 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:02:29.863764 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:02:29.863731 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:31.724149 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:02:31.724103 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:34.106645 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:02:34.106603 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:35.522429 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:02:35.522366 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:36.730105 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:02:36.730059 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:37.569601 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:02:37.569564 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:38.234718 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:02:38.234681 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:38.541671 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:02:38.541582 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:41.977934 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:02:41.977895 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:42.289907 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:02:42.289870 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:42.523477 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:02:42.523438 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:45.437480 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:02:45.437446 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:45.656229 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:02:45.656189 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:47.526415 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:02:47.526378 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:52.518952 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:02:52.518915 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:57.518041 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:02:57.518003 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:02:58.472768 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:02:58.472732 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:03:02.477242 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:02.477205 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:07.535759 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:07.535721 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:09.605087 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:09.605048 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:12.545792 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:12.545753 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:15.630678 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:15.630633 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:17.536833 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:17.536793 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:22.480748 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:22.480710 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:27.010686 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:27.010642 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq"] Nov 02 22:03:27.139801 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:27.139758 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:27.463574 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:27.463538 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:28.401853 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:28.401821 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 22:03:28.416252 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:28.416224 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 22:03:28.424072 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:28.424044 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 22:03:28.435061 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:28.435039 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 22:03:28.470542 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:03:28.470509 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:03:32.436484 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:32.436435 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:35.863810 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:35.863769 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:37.547646 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:37.547604 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:39.398314 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:39.398278 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:39.943154 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:39.942824 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:40.170173 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:40.170110 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:42.032093 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:42.032035 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" podUID="f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a" containerName="main" containerID="cri-o://cdc6edde1e3370cd4026533b6075a6b4069f7f59fd2fd56462771ce8b8100638" gracePeriod=45 Nov 02 22:03:42.222964 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:42.222875 4935 generic.go:356] "Generic (PLEG): container finished" podID="f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a" containerID="cdc6edde1e3370cd4026533b6075a6b4069f7f59fd2fd56462771ce8b8100638" exitCode=0 Nov 02 22:03:42.223148 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:42.222950 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" event={"ID":"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a","Type":"ContainerDied","Data":"cdc6edde1e3370cd4026533b6075a6b4069f7f59fd2fd56462771ce8b8100638"} Nov 02 22:03:42.313701 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:42.313673 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" Nov 02 22:03:42.381920 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:42.381884 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-model-cache\") pod \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\" (UID: \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\") " Nov 02 22:03:42.382103 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:42.381940 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-home\" (UniqueName: \"kubernetes.io/empty-dir/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-home\") pod \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\" (UID: \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\") " Nov 02 22:03:42.382103 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:42.381973 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-tls-certs\" (UniqueName: \"kubernetes.io/secret/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-tls-certs\") pod \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\" (UID: \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\") " Nov 02 22:03:42.382103 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:42.382013 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-dshm\") pod \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\" (UID: \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\") " Nov 02 22:03:42.382103 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:42.382038 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-tmp-dir\") pod \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\" (UID: \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\") " Nov 02 22:03:42.382373 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:42.382176 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-kube-api-access-nj5lb\" (UniqueName: \"kubernetes.io/projected/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-kube-api-access-nj5lb\") pod \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\" (UID: \"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a\") " Nov 02 22:03:42.382373 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:42.382237 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-home" pod "f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a" (UID: "f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:03:42.382373 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:42.382251 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-model-cache" pod "f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a" (UID: "f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:03:42.382373 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:42.382334 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-tmp-dir" pod "f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a" (UID: "f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:03:42.382630 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:42.382612 4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-model-cache\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:42.382717 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:42.382632 4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-home\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:42.382717 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:42.382641 4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-tmp-dir\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:42.384387 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:42.384363 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-tls-certs" pod "f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a" (UID: "f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:03:42.384468 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:42.384402 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-dshm" pod "f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a" (UID: "f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:03:42.384468 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:42.384430 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-kube-api-access-nj5lb" pod "f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a" (UID: "f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a"). InnerVolumeSpecName "kube-api-access-nj5lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:03:42.484023 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:42.483981 4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-tls-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:42.484023 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:42.484015 4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-dshm\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:42.484023 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:42.484027 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj5lb\" (UniqueName: \"kubernetes.io/projected/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a-kube-api-access-nj5lb\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:42.609155 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:42.609046 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:43.233210 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:43.233171 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" event={"ID":"f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a","Type":"ContainerDied","Data":"6b391f7322167c5e2b7663b8f6cf19f50c29cf115189594b6b25cbb19b1938ee"} Nov 02 22:03:43.233679 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:43.233231 4935 scope.go:122] "RemoveContainer" containerID="cdc6edde1e3370cd4026533b6075a6b4069f7f59fd2fd56462771ce8b8100638" Nov 02 22:03:43.233679 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:43.233298 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq" Nov 02 22:03:43.386539 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:43.386501 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq"] Nov 02 22:03:43.432289 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:43.432250 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a1d751/auth-test-model-a1d751-kserve-6fd8ffc557-mhgbq"] Nov 02 22:03:43.684586 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:43.684549 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:43.776629 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:43.776595 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a" path="/var/lib/kubelet/pods/f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a/volumes" Nov 02 22:03:43.913844 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:43.913809 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:45.745733 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:45.745689 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:46.823403 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:46.823359 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:47.041707 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:47.041671 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:48.135087 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:48.135053 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:53.180210 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:53.180176 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:53.393882 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:53.393847 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:58.426441 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:03:58.426404 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:03:58.470283 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:03:58.470243 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:04:03.469800 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:03.469758 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:08.455687 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:08.455647 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:13.421095 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:13.421057 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:18.527738 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:18.527700 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:23.495880 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:23.495841 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:28.469755 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:04:28.469715 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:04:28.541836 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:28.541792 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:33.578936 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:33.578898 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:36.120237 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:36.120198 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl"] Nov 02 22:04:36.221224 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:36.221062 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:36.447975 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:36.447881 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:38.554827 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:38.554735 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:43.561501 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:43.561446 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:48.642325 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:48.642289 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:51.141448 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.141389 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" podUID="21877724-6647-4d7a-ae34-d01a1da09ed8" containerName="main" containerID="cri-o://823ccb1037d0caf791301dd4ed564081ae370b1b829567ae117d927157c8fdeb" gracePeriod=45 Nov 02 22:04:51.414357 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.414327 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" Nov 02 22:04:51.486419 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.486373 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21877724-6647-4d7a-ae34-d01a1da09ed8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21877724-6647-4d7a-ae34-d01a1da09ed8-model-cache\") pod \"21877724-6647-4d7a-ae34-d01a1da09ed8\" (UID: \"21877724-6647-4d7a-ae34-d01a1da09ed8\") " Nov 02 22:04:51.486596 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.486504 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21877724-6647-4d7a-ae34-d01a1da09ed8-tls-certs\" (UniqueName: \"kubernetes.io/secret/21877724-6647-4d7a-ae34-d01a1da09ed8-tls-certs\") pod \"21877724-6647-4d7a-ae34-d01a1da09ed8\" (UID: \"21877724-6647-4d7a-ae34-d01a1da09ed8\") " Nov 02 22:04:51.486596 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.486532 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21877724-6647-4d7a-ae34-d01a1da09ed8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21877724-6647-4d7a-ae34-d01a1da09ed8-tmp-dir\") pod \"21877724-6647-4d7a-ae34-d01a1da09ed8\" (UID: \"21877724-6647-4d7a-ae34-d01a1da09ed8\") " Nov 02 22:04:51.486596 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.486572 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21877724-6647-4d7a-ae34-d01a1da09ed8-home\" (UniqueName: \"kubernetes.io/empty-dir/21877724-6647-4d7a-ae34-d01a1da09ed8-home\") pod \"21877724-6647-4d7a-ae34-d01a1da09ed8\" (UID: \"21877724-6647-4d7a-ae34-d01a1da09ed8\") " Nov 02 22:04:51.486596 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.486593 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21877724-6647-4d7a-ae34-d01a1da09ed8-kube-api-access-v8sf4\" (UniqueName: \"kubernetes.io/projected/21877724-6647-4d7a-ae34-d01a1da09ed8-kube-api-access-v8sf4\") pod \"21877724-6647-4d7a-ae34-d01a1da09ed8\" (UID: \"21877724-6647-4d7a-ae34-d01a1da09ed8\") " Nov 02 22:04:51.486847 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.486625 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21877724-6647-4d7a-ae34-d01a1da09ed8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/21877724-6647-4d7a-ae34-d01a1da09ed8-dshm\") pod \"21877724-6647-4d7a-ae34-d01a1da09ed8\" (UID: \"21877724-6647-4d7a-ae34-d01a1da09ed8\") " Nov 02 22:04:51.486847 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.486649 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21877724-6647-4d7a-ae34-d01a1da09ed8-model-cache" pod "21877724-6647-4d7a-ae34-d01a1da09ed8" (UID: "21877724-6647-4d7a-ae34-d01a1da09ed8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:51.486847 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.486816 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21877724-6647-4d7a-ae34-d01a1da09ed8-home" pod "21877724-6647-4d7a-ae34-d01a1da09ed8" (UID: "21877724-6647-4d7a-ae34-d01a1da09ed8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:51.486992 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.486850 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21877724-6647-4d7a-ae34-d01a1da09ed8-tmp-dir" pod "21877724-6647-4d7a-ae34-d01a1da09ed8" (UID: "21877724-6647-4d7a-ae34-d01a1da09ed8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:51.487040 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.487026 4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21877724-6647-4d7a-ae34-d01a1da09ed8-model-cache\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:51.487105 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.487040 4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21877724-6647-4d7a-ae34-d01a1da09ed8-tmp-dir\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:51.487105 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.487050 4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21877724-6647-4d7a-ae34-d01a1da09ed8-home\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:51.488955 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.488927 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21877724-6647-4d7a-ae34-d01a1da09ed8-dshm" pod "21877724-6647-4d7a-ae34-d01a1da09ed8" (UID: "21877724-6647-4d7a-ae34-d01a1da09ed8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:51.489064 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.488966 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21877724-6647-4d7a-ae34-d01a1da09ed8-kube-api-access-v8sf4" pod "21877724-6647-4d7a-ae34-d01a1da09ed8" (UID: "21877724-6647-4d7a-ae34-d01a1da09ed8"). InnerVolumeSpecName "kube-api-access-v8sf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:51.489064 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.488975 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21877724-6647-4d7a-ae34-d01a1da09ed8-tls-certs" pod "21877724-6647-4d7a-ae34-d01a1da09ed8" (UID: "21877724-6647-4d7a-ae34-d01a1da09ed8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:04:51.588587 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.588544 4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21877724-6647-4d7a-ae34-d01a1da09ed8-tls-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:51.588587 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.588587 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8sf4\" (UniqueName: \"kubernetes.io/projected/21877724-6647-4d7a-ae34-d01a1da09ed8-kube-api-access-v8sf4\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:51.588797 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.588602 4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21877724-6647-4d7a-ae34-d01a1da09ed8-dshm\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:51.691278 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.691183 4935 generic.go:356] "Generic (PLEG): container finished" podID="21877724-6647-4d7a-ae34-d01a1da09ed8" containerID="823ccb1037d0caf791301dd4ed564081ae370b1b829567ae117d927157c8fdeb" exitCode=0 Nov 02 22:04:51.691278 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.691233 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" Nov 02 22:04:51.691278 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.691266 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" event={"ID":"21877724-6647-4d7a-ae34-d01a1da09ed8","Type":"ContainerDied","Data":"823ccb1037d0caf791301dd4ed564081ae370b1b829567ae117d927157c8fdeb"} Nov 02 22:04:51.691570 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.691313 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl" event={"ID":"21877724-6647-4d7a-ae34-d01a1da09ed8","Type":"ContainerDied","Data":"54998987f00ad9dce8ba837e0979f480892a73587df2e469c3d44da60cd42d80"} Nov 02 22:04:51.691570 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.691331 4935 scope.go:122] "RemoveContainer" containerID="823ccb1037d0caf791301dd4ed564081ae370b1b829567ae117d927157c8fdeb" Nov 02 22:04:51.707408 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.707387 4935 scope.go:122] "RemoveContainer" containerID="823ccb1037d0caf791301dd4ed564081ae370b1b829567ae117d927157c8fdeb" Nov 02 22:04:51.707753 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:04:51.707727 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"823ccb1037d0caf791301dd4ed564081ae370b1b829567ae117d927157c8fdeb\": container with ID starting with 823ccb1037d0caf791301dd4ed564081ae370b1b829567ae117d927157c8fdeb not found: ID does not exist" containerID="823ccb1037d0caf791301dd4ed564081ae370b1b829567ae117d927157c8fdeb" Nov 02 22:04:51.707847 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.707761 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"823ccb1037d0caf791301dd4ed564081ae370b1b829567ae117d927157c8fdeb"} err="failed to get container status \"823ccb1037d0caf791301dd4ed564081ae370b1b829567ae117d927157c8fdeb\": rpc error: code = NotFound desc = could not find container \"823ccb1037d0caf791301dd4ed564081ae370b1b829567ae117d927157c8fdeb\": container with ID starting with 823ccb1037d0caf791301dd4ed564081ae370b1b829567ae117d927157c8fdeb not found: ID does not exist" Nov 02 22:04:51.844449 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.844415 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl"] Nov 02 22:04:51.849017 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:51.848988 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6e535d/auth-test-model-6e535d-kserve-5489cb5f7-z2kfl"] Nov 02 22:04:53.635251 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:53.635206 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:53.778236 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:53.778203 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21877724-6647-4d7a-ae34-d01a1da09ed8" path="/var/lib/kubelet/pods/21877724-6647-4d7a-ae34-d01a1da09ed8/volumes" Nov 02 22:04:53.851262 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:53.851220 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:58.222535 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.222495 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7"] Nov 02 22:04:58.223273 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223251 4935 cpu_manager.go:414] "containerMap: removing container" podUID="297af528-36c0-423d-9bb4-0cbe3bec1bd3" containerName="maas-api" Nov 02 22:04:58.223401 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223275 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="297af528-36c0-423d-9bb4-0cbe3bec1bd3" containerName="maas-api" Nov 02 22:04:58.223401 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223293 4935 cpu_manager.go:414] "containerMap: removing container" podUID="f7c556b2-03fd-4529-9046-a385270e2d13" containerName="cleanup" Nov 02 22:04:58.223401 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223302 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7c556b2-03fd-4529-9046-a385270e2d13" containerName="cleanup" Nov 02 22:04:58.223401 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223324 4935 cpu_manager.go:414] "containerMap: removing container" podUID="21877724-6647-4d7a-ae34-d01a1da09ed8" containerName="main" Nov 02 22:04:58.223401 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223332 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21877724-6647-4d7a-ae34-d01a1da09ed8" containerName="main" Nov 02 22:04:58.223401 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223349 4935 cpu_manager.go:414] "containerMap: removing container" podUID="f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a" containerName="main" Nov 02 22:04:58.223401 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223357 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a" containerName="main" Nov 02 22:04:58.223401 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223381 4935 cpu_manager.go:414] "containerMap: removing container" podUID="9cb8d510-bb7b-4598-886b-c26a6f891576" containerName="cleanup" Nov 02 22:04:58.223401 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223388 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cb8d510-bb7b-4598-886b-c26a6f891576" containerName="cleanup" Nov 02 22:04:58.223929 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223414 4935 cpu_manager.go:414] "containerMap: removing container" podUID="83930246-18af-44a1-b4c6-3353bbba6814" containerName="istio-proxy" Nov 02 22:04:58.223929 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223422 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83930246-18af-44a1-b4c6-3353bbba6814" containerName="istio-proxy" Nov 02 22:04:58.223929 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223436 4935 cpu_manager.go:414] "containerMap: removing container" podUID="f7c556b2-03fd-4529-9046-a385270e2d13" containerName="cleanup" Nov 02 22:04:58.223929 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223442 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7c556b2-03fd-4529-9046-a385270e2d13" containerName="cleanup" Nov 02 22:04:58.223929 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223455 4935 cpu_manager.go:414] "containerMap: removing container" podUID="9cb8d510-bb7b-4598-886b-c26a6f891576" containerName="cleanup" Nov 02 22:04:58.223929 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223462 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cb8d510-bb7b-4598-886b-c26a6f891576" containerName="cleanup" Nov 02 22:04:58.223929 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223472 4935 cpu_manager.go:414] "containerMap: removing container" podUID="9cb8d510-bb7b-4598-886b-c26a6f891576" containerName="cleanup" Nov 02 22:04:58.223929 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223480 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cb8d510-bb7b-4598-886b-c26a6f891576" containerName="cleanup" Nov 02 22:04:58.223929 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223491 4935 cpu_manager.go:414] "containerMap: removing container" podUID="f7c556b2-03fd-4529-9046-a385270e2d13" containerName="cleanup" Nov 02 22:04:58.223929 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223498 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7c556b2-03fd-4529-9046-a385270e2d13" containerName="cleanup" Nov 02 22:04:58.223929 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223627 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cb8d510-bb7b-4598-886b-c26a6f891576" containerName="cleanup" Nov 02 22:04:58.223929 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223646 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="83930246-18af-44a1-b4c6-3353bbba6814" containerName="istio-proxy" Nov 02 22:04:58.223929 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223654 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7c556b2-03fd-4529-9046-a385270e2d13" containerName="cleanup" Nov 02 22:04:58.223929 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223663 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cb8d510-bb7b-4598-886b-c26a6f891576" containerName="cleanup" Nov 02 22:04:58.223929 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223672 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cb8d510-bb7b-4598-886b-c26a6f891576" containerName="cleanup" Nov 02 22:04:58.223929 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223688 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="297af528-36c0-423d-9bb4-0cbe3bec1bd3" containerName="maas-api" Nov 02 22:04:58.223929 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223699 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7c556b2-03fd-4529-9046-a385270e2d13" containerName="cleanup" Nov 02 22:04:58.223929 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223710 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="21877724-6647-4d7a-ae34-d01a1da09ed8" containerName="main" Nov 02 22:04:58.223929 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.223719 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7e3a7a2-ba1b-4dcd-b9aa-43f22f44398a" containerName="main" Nov 02 22:04:58.228796 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.228774 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7" Nov 02 22:04:58.266940 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.266899 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkk9c\" (UniqueName: \"kubernetes.io/projected/ec842943-f0f8-4fd0-b32d-96314b30077f-kube-api-access-tkk9c\") pod \"maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7\" (UID: \"ec842943-f0f8-4fd0-b32d-96314b30077f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7" Nov 02 22:04:58.267122 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.267005 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec842943-f0f8-4fd0-b32d-96314b30077f-metrics-certs\") pod \"maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7\" (UID: \"ec842943-f0f8-4fd0-b32d-96314b30077f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7" Nov 02 22:04:58.267122 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.267063 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec842943-f0f8-4fd0-b32d-96314b30077f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7\" (UID: \"ec842943-f0f8-4fd0-b32d-96314b30077f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7" Nov 02 22:04:58.267122 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.267093 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec842943-f0f8-4fd0-b32d-96314b30077f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7\" (UID: \"ec842943-f0f8-4fd0-b32d-96314b30077f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7" Nov 02 22:04:58.277565 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.277535 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-8702a9-serving-cert\"" Nov 02 22:04:58.326578 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.326545 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7"] Nov 02 22:04:58.368494 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.368452 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec842943-f0f8-4fd0-b32d-96314b30077f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7\" (UID: \"ec842943-f0f8-4fd0-b32d-96314b30077f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7" Nov 02 22:04:58.368775 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.368749 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec842943-f0f8-4fd0-b32d-96314b30077f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7\" (UID: \"ec842943-f0f8-4fd0-b32d-96314b30077f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7" Nov 02 22:04:58.369214 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.369188 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkk9c\" (UniqueName: \"kubernetes.io/projected/ec842943-f0f8-4fd0-b32d-96314b30077f-kube-api-access-tkk9c\") pod \"maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7\" (UID: \"ec842943-f0f8-4fd0-b32d-96314b30077f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7" Nov 02 22:04:58.369396 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.369379 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec842943-f0f8-4fd0-b32d-96314b30077f-metrics-certs\") pod \"maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7\" (UID: \"ec842943-f0f8-4fd0-b32d-96314b30077f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7" Nov 02 22:04:58.369553 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.369431 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec842943-f0f8-4fd0-b32d-96314b30077f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7\" (UID: \"ec842943-f0f8-4fd0-b32d-96314b30077f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7" Nov 02 22:04:58.369670 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:04:58.368970 4935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-serving-cert: secret "maas-api-e2e-extmod-a-8702a9-serving-cert" not found Nov 02 22:04:58.369842 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:04:58.369829 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec842943-f0f8-4fd0-b32d-96314b30077f-maas-api-tls podName:ec842943-f0f8-4fd0-b32d-96314b30077f nodeName:}" failed. No retries permitted until 2026-11-02 22:04:58.869806961 +0000 UTC m=+3391.309381664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ec842943-f0f8-4fd0-b32d-96314b30077f-maas-api-tls") pod "maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7" (UID: "ec842943-f0f8-4fd0-b32d-96314b30077f") : secret "maas-api-e2e-extmod-a-8702a9-serving-cert" not found Nov 02 22:04:58.386623 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.386598 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec842943-f0f8-4fd0-b32d-96314b30077f-metrics-certs\") pod \"maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7\" (UID: \"ec842943-f0f8-4fd0-b32d-96314b30077f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7" Nov 02 22:04:58.423620 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.423582 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkk9c\" (UniqueName: \"kubernetes.io/projected/ec842943-f0f8-4fd0-b32d-96314b30077f-kube-api-access-tkk9c\") pod \"maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7\" (UID: \"ec842943-f0f8-4fd0-b32d-96314b30077f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7" Nov 02 22:04:58.470242 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:04:58.470197 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:04:58.510580 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.510500 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:58.729077 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.729037 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:04:58.875588 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.875557 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec842943-f0f8-4fd0-b32d-96314b30077f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7\" (UID: \"ec842943-f0f8-4fd0-b32d-96314b30077f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7" Nov 02 22:04:58.878167 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:58.878142 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec842943-f0f8-4fd0-b32d-96314b30077f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7\" (UID: \"ec842943-f0f8-4fd0-b32d-96314b30077f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7" Nov 02 22:04:59.145010 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:59.144923 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7" Nov 02 22:04:59.517415 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:59.517385 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7"] Nov 02 22:04:59.519678 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:04:59.519645 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec842943_f0f8_4fd0_b32d_96314b30077f.slice/crio-09f3d06efa2cbd0be2ad5957d9ffa90d5011c6069e12b6bd5fc0b871ef9c19f7 WatchSource:0}: Error finding container 09f3d06efa2cbd0be2ad5957d9ffa90d5011c6069e12b6bd5fc0b871ef9c19f7: Status 404 returned error can't find the container with id 09f3d06efa2cbd0be2ad5957d9ffa90d5011c6069e12b6bd5fc0b871ef9c19f7 Nov 02 22:04:59.521094 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:59.521075 4935 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:04:59.742103 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:04:59.742061 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7" event={"ID":"ec842943-f0f8-4fd0-b32d-96314b30077f","Type":"ContainerStarted","Data":"09f3d06efa2cbd0be2ad5957d9ffa90d5011c6069e12b6bd5fc0b871ef9c19f7"} Nov 02 22:05:00.751225 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:00.751193 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7" event={"ID":"ec842943-f0f8-4fd0-b32d-96314b30077f","Type":"ContainerStarted","Data":"d480510702914ea1cd9e9fb1b8cc72c1de5ba89a5f3c2d6fc08021faec521427"} Nov 02 22:05:00.751620 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:00.751252 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7" Nov 02 22:05:00.859614 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:00.859504 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7" podStartSLOduration=1.846034077 podStartE2EDuration="2.859486746s" totalImagesPullingTime="1.013452669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:00.856236382 +0000 UTC m=+3393.295811104" watchObservedRunningTime="2026-11-02 22:05:00.859486746 +0000 UTC m=+3393.299061509" Nov 02 22:05:03.743239 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:03.743152 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:06.764009 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:06.763980 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7" Nov 02 22:05:08.335071 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:08.335014 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:08.775955 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:08.775918 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:09.329816 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:09.329484 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:09.546065 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:09.546026 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:12.816036 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:12.816004 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:13.124966 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:13.124890 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:14.151001 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:14.150956 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:15.947258 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:15.947214 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:16.168798 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:16.168748 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:19.341650 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:19.341607 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:20.671524 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:20.671477 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:21.110104 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.110060 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:21.719083 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.719050 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94"] Nov 02 22:05:21.719678 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.719661 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7c556b2-03fd-4529-9046-a385270e2d13" containerName="cleanup" Nov 02 22:05:21.723220 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.723198 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:21.782685 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.782648 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-5ddbd0-openshift-default-dockercfg-xb4xp\"" Nov 02 22:05:21.833812 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.833748 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94"] Nov 02 22:05:21.842006 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.841903 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istiod-ca-cert\") pod \"e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:21.842329 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.842303 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9xrs\" (UniqueName: \"kubernetes.io/projected/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-kube-api-access-k9xrs\") pod \"e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:21.842558 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.842540 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istio-data\") pod \"e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:21.842674 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.842660 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istio-podinfo\") pod \"e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:21.842766 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.842753 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istio-token\") pod \"e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:21.842889 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.842875 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-credential-socket\") pod \"e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:21.843082 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.843066 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istio-envoy\") pod \"e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:21.843250 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.843236 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-workload-certs\") pod \"e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:21.843365 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.843352 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-workload-socket\") pod \"e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:21.936199 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.936126 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:21.945432 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.945396 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istio-data\") pod \"e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:21.945432 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.945431 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istio-podinfo\") pod \"e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:21.945642 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.945447 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istio-token\") pod \"e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:21.945642 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.945464 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-credential-socket\") pod \"e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:21.945642 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.945525 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istio-envoy\") pod \"e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:21.945642 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.945568 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-workload-certs\") pod \"e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:21.945642 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.945597 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-workload-socket\") pod \"e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:21.945915 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.945678 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istiod-ca-cert\") pod \"e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:21.945915 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.945734 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9xrs\" (UniqueName: \"kubernetes.io/projected/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-kube-api-access-k9xrs\") pod \"e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:21.945915 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.945875 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istio-data\") pod \"e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:21.946267 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.946241 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-workload-socket\") pod \"e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:21.946539 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.946516 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-workload-certs\") pod \"e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:21.946776 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.946756 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-credential-socket\") pod \"e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:21.946841 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.946777 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istiod-ca-cert\") pod \"e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:21.948504 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.948481 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istio-envoy\") pod \"e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:21.948828 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:21.948812 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istio-podinfo\") pod \"e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:22.000548 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:22.000468 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9xrs\" (UniqueName: \"kubernetes.io/projected/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-kube-api-access-k9xrs\") pod \"e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:22.001717 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:22.001686 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istio-token\") pod \"e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:22.037715 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:22.037649 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:22.154703 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:22.154606 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:22.462856 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:22.462383 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94"] Nov 02 22:05:22.465737 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:05:22.465700 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e7ef62_e7ba_474a_9f7d_650b4c98d0de.slice/crio-9765864d4eb116fc3bb93cf896dab9afff13c551c27654322cbcaab46604e2a9 WatchSource:0}: Error finding container 9765864d4eb116fc3bb93cf896dab9afff13c551c27654322cbcaab46604e2a9: Status 404 returned error can't find the container with id 9765864d4eb116fc3bb93cf896dab9afff13c551c27654322cbcaab46604e2a9 Nov 02 22:05:22.469617 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:22.469574 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:05:22.469765 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:22.469669 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:05:22.469765 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:22.469715 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:05:22.902992 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:22.902954 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" event={"ID":"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de","Type":"ContainerStarted","Data":"aa31aa5acabd4a99eda1990486f703252f925574e10cc265cd0e1840b9735aa2"} Nov 02 22:05:22.903535 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:22.903009 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" event={"ID":"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de","Type":"ContainerStarted","Data":"9765864d4eb116fc3bb93cf896dab9afff13c551c27654322cbcaab46604e2a9"} Nov 02 22:05:23.016036 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:23.015972 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" podStartSLOduration=2.015951273 podStartE2EDuration="2.015951273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:23.01027962 +0000 UTC m=+3415.449854357" watchObservedRunningTime="2026-11-02 22:05:23.015951273 +0000 UTC m=+3415.455525999" Nov 02 22:05:23.038353 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:23.038324 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:23.045872 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:23.045840 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:23.909431 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:23.909399 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:23.910421 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:23.910400 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:05:24.838276 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:24.838244 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:25.927342 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:25.927305 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:26.171833 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:26.171790 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:28.469593 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:05:28.469557 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:05:29.168659 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:29.168576 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:29.386506 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:29.386457 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:30.441171 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:30.441110 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:31.478878 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:31.478824 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:35.416552 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:35.416504 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:35.908713 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:35.908676 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw"] Nov 02 22:05:36.003823 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:36.003786 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:36.550667 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:36.550630 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:38.801672 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:38.801630 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:40.314499 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:40.314453 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:40.542180 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:40.542116 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:40.943189 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:40.943112 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:41.486253 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:41.486213 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:45.183488 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:45.183451 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:45.409585 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:45.409548 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:45.969982 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:45.969946 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:46.194624 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:46.194578 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:48.635635 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:48.635595 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:48.852877 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:48.852835 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:50.931017 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:50.930950 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" podUID="59e43da6-a639-4412-a666-93c5d8b04120" containerName="main" containerID="cri-o://e7797e3b1b7100ceee1ea6e8a85b1378b405a159136cd4b15a785c6f0923f57a" gracePeriod=45 Nov 02 22:05:51.122386 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:51.122166 4935 generic.go:356] "Generic (PLEG): container finished" podID="59e43da6-a639-4412-a666-93c5d8b04120" containerID="e7797e3b1b7100ceee1ea6e8a85b1378b405a159136cd4b15a785c6f0923f57a" exitCode=0 Nov 02 22:05:51.122386 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:51.122186 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" event={"ID":"59e43da6-a639-4412-a666-93c5d8b04120","Type":"ContainerDied","Data":"e7797e3b1b7100ceee1ea6e8a85b1378b405a159136cd4b15a785c6f0923f57a"} Nov 02 22:05:51.191590 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:51.191526 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" Nov 02 22:05:51.326082 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:51.326046 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59e43da6-a639-4412-a666-93c5d8b04120-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59e43da6-a639-4412-a666-93c5d8b04120-tmp-dir\") pod \"59e43da6-a639-4412-a666-93c5d8b04120\" (UID: \"59e43da6-a639-4412-a666-93c5d8b04120\") " Nov 02 22:05:51.326304 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:51.326120 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59e43da6-a639-4412-a666-93c5d8b04120-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59e43da6-a639-4412-a666-93c5d8b04120-model-cache\") pod \"59e43da6-a639-4412-a666-93c5d8b04120\" (UID: \"59e43da6-a639-4412-a666-93c5d8b04120\") " Nov 02 22:05:51.326304 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:51.326193 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59e43da6-a639-4412-a666-93c5d8b04120-tls-certs\" (UniqueName: \"kubernetes.io/secret/59e43da6-a639-4412-a666-93c5d8b04120-tls-certs\") pod \"59e43da6-a639-4412-a666-93c5d8b04120\" (UID: \"59e43da6-a639-4412-a666-93c5d8b04120\") " Nov 02 22:05:51.326304 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:51.326263 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59e43da6-a639-4412-a666-93c5d8b04120-home\" (UniqueName: \"kubernetes.io/empty-dir/59e43da6-a639-4412-a666-93c5d8b04120-home\") pod \"59e43da6-a639-4412-a666-93c5d8b04120\" (UID: \"59e43da6-a639-4412-a666-93c5d8b04120\") " Nov 02 22:05:51.326484 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:51.326358 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59e43da6-a639-4412-a666-93c5d8b04120-model-cache" pod "59e43da6-a639-4412-a666-93c5d8b04120" (UID: "59e43da6-a639-4412-a666-93c5d8b04120"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:05:51.326484 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:51.326396 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59e43da6-a639-4412-a666-93c5d8b04120-kube-api-access-t4spk\" (UniqueName: \"kubernetes.io/projected/59e43da6-a639-4412-a666-93c5d8b04120-kube-api-access-t4spk\") pod \"59e43da6-a639-4412-a666-93c5d8b04120\" (UID: \"59e43da6-a639-4412-a666-93c5d8b04120\") " Nov 02 22:05:51.326484 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:51.326452 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59e43da6-a639-4412-a666-93c5d8b04120-dshm\" (UniqueName: \"kubernetes.io/empty-dir/59e43da6-a639-4412-a666-93c5d8b04120-dshm\") pod \"59e43da6-a639-4412-a666-93c5d8b04120\" (UID: \"59e43da6-a639-4412-a666-93c5d8b04120\") " Nov 02 22:05:51.326969 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:51.326945 4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59e43da6-a639-4412-a666-93c5d8b04120-model-cache\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:51.327097 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:51.327013 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59e43da6-a639-4412-a666-93c5d8b04120-home" pod "59e43da6-a639-4412-a666-93c5d8b04120" (UID: "59e43da6-a639-4412-a666-93c5d8b04120"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:05:51.327370 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:51.327341 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59e43da6-a639-4412-a666-93c5d8b04120-tmp-dir" pod "59e43da6-a639-4412-a666-93c5d8b04120" (UID: "59e43da6-a639-4412-a666-93c5d8b04120"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:05:51.330299 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:51.330271 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59e43da6-a639-4412-a666-93c5d8b04120-tls-certs" pod "59e43da6-a639-4412-a666-93c5d8b04120" (UID: "59e43da6-a639-4412-a666-93c5d8b04120"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:05:51.330408 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:51.330296 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59e43da6-a639-4412-a666-93c5d8b04120-kube-api-access-t4spk" pod "59e43da6-a639-4412-a666-93c5d8b04120" (UID: "59e43da6-a639-4412-a666-93c5d8b04120"). InnerVolumeSpecName "kube-api-access-t4spk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:05:51.330408 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:51.330372 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59e43da6-a639-4412-a666-93c5d8b04120-dshm" pod "59e43da6-a639-4412-a666-93c5d8b04120" (UID: "59e43da6-a639-4412-a666-93c5d8b04120"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:05:51.428397 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:51.428366 4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59e43da6-a639-4412-a666-93c5d8b04120-home\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:51.428397 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:51.428394 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4spk\" (UniqueName: \"kubernetes.io/projected/59e43da6-a639-4412-a666-93c5d8b04120-kube-api-access-t4spk\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:51.428397 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:51.428403 4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59e43da6-a639-4412-a666-93c5d8b04120-dshm\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:51.428631 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:51.428412 4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59e43da6-a639-4412-a666-93c5d8b04120-tmp-dir\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:51.428631 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:51.428428 4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59e43da6-a639-4412-a666-93c5d8b04120-tls-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:51.636296 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:51.636260 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:52.131064 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:52.131030 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" Nov 02 22:05:52.131597 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:52.131050 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw" event={"ID":"59e43da6-a639-4412-a666-93c5d8b04120","Type":"ContainerDied","Data":"d425066f7835377fbd0e43e840b5cb3c9e2b7a1bc117e3ea4962983ea32fb77a"} Nov 02 22:05:52.131597 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:52.131116 4935 scope.go:122] "RemoveContainer" containerID="e7797e3b1b7100ceee1ea6e8a85b1378b405a159136cd4b15a785c6f0923f57a" Nov 02 22:05:52.284560 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:52.284523 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw"] Nov 02 22:05:52.335161 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:52.331846 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-0f0874/auth-test-model-0f0874-kserve-79888dfff7-ggjhw"] Nov 02 22:05:53.777343 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:53.777311 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59e43da6-a639-4412-a666-93c5d8b04120" path="/var/lib/kubelet/pods/59e43da6-a639-4412-a666-93c5d8b04120/volumes" Nov 02 22:05:56.742374 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:05:56.742336 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:58.469648 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:05:58.469607 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:06:01.928455 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:01.928419 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:06.839868 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:06.839834 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:11.849723 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:11.849636 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:16.855263 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:16.855219 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:21.846732 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:21.846689 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:26.924925 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:26.924884 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:28.471774 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:06:28.471731 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:06:32.223382 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:32.223345 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:32.452856 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:32.452824 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:33.482584 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:33.482547 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:33.871375 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:33.871333 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq"] Nov 02 22:06:33.872150 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:33.872115 4935 cpu_manager.go:414] "containerMap: removing container" podUID="59e43da6-a639-4412-a666-93c5d8b04120" containerName="main" Nov 02 22:06:33.872249 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:33.872152 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59e43da6-a639-4412-a666-93c5d8b04120" containerName="main" Nov 02 22:06:33.872318 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:33.872306 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="59e43da6-a639-4412-a666-93c5d8b04120" containerName="main" Nov 02 22:06:33.875689 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:33.875669 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:33.977152 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:33.975112 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq"] Nov 02 22:06:33.977152 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:33.975441 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-6ba06f-openshift-default-dockercfg-7d5kb\"" Nov 02 22:06:33.985815 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:33.985675 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-workload-certs\") pod \"e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:33.986168 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:33.986124 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf1a756-7259-404a-aa43-ca017939d6ec-istio-token\") pod \"e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:33.986345 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:33.986322 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-istio-envoy\") pod \"e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:33.986472 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:33.986457 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2fhq\" (UniqueName: \"kubernetes.io/projected/8bf1a756-7259-404a-aa43-ca017939d6ec-kube-api-access-d2fhq\") pod \"e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:33.986730 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:33.986712 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-credential-socket\") pod \"e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:33.988694 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:33.988510 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-workload-socket\") pod \"e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:33.989150 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:33.989116 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf1a756-7259-404a-aa43-ca017939d6ec-istiod-ca-cert\") pod \"e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:33.989323 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:33.989302 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-istio-data\") pod \"e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:33.989486 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:33.989455 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf1a756-7259-404a-aa43-ca017939d6ec-istio-podinfo\") pod \"e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:34.012979 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:34.012930 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:34.090933 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:34.090890 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-workload-socket\") pod \"e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:34.091171 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:34.090973 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf1a756-7259-404a-aa43-ca017939d6ec-istiod-ca-cert\") pod \"e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:34.091171 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:34.090999 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-istio-data\") pod \"e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:34.091171 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:34.091019 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf1a756-7259-404a-aa43-ca017939d6ec-istio-podinfo\") pod \"e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:34.091171 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:34.091058 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-workload-certs\") pod \"e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:34.091171 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:34.091087 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf1a756-7259-404a-aa43-ca017939d6ec-istio-token\") pod \"e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:34.091171 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:34.091113 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-istio-envoy\") pod \"e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:34.091171 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:34.091153 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2fhq\" (UniqueName: \"kubernetes.io/projected/8bf1a756-7259-404a-aa43-ca017939d6ec-kube-api-access-d2fhq\") pod \"e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:34.091563 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:34.091255 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-credential-socket\") pod \"e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:34.091563 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:34.091486 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-workload-socket\") pod \"e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:34.091563 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:34.091502 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-istio-data\") pod \"e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:34.091563 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:34.091552 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-credential-socket\") pod \"e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:34.091809 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:34.091783 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf1a756-7259-404a-aa43-ca017939d6ec-istiod-ca-cert\") pod \"e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:34.092079 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:34.092054 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-workload-certs\") pod \"e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:34.094010 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:34.093986 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-istio-envoy\") pod \"e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:34.094119 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:34.094012 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf1a756-7259-404a-aa43-ca017939d6ec-istio-podinfo\") pod \"e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:34.146227 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:34.146157 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf1a756-7259-404a-aa43-ca017939d6ec-istio-token\") pod \"e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:34.146566 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:34.146533 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2fhq\" (UniqueName: \"kubernetes.io/projected/8bf1a756-7259-404a-aa43-ca017939d6ec-kube-api-access-d2fhq\") pod \"e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:34.192707 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:34.192666 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:34.247412 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:34.247354 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:34.466201 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:34.465810 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:34.628758 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:34.628731 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq"] Nov 02 22:06:34.630821 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:06:34.630785 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf1a756_7259_404a_aa43_ca017939d6ec.slice/crio-255e3fe4b9ff6e2a1dbc46e4461cf9dc398ef1a6bf8727338d3b61f127c3efc2 WatchSource:0}: Error finding container 255e3fe4b9ff6e2a1dbc46e4461cf9dc398ef1a6bf8727338d3b61f127c3efc2: Status 404 returned error can't find the container with id 255e3fe4b9ff6e2a1dbc46e4461cf9dc398ef1a6bf8727338d3b61f127c3efc2 Nov 02 22:06:34.633550 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:34.633513 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:06:34.633667 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:34.633585 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:06:34.633667 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:34.633615 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:06:35.409293 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:35.409245 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" event={"ID":"8bf1a756-7259-404a-aa43-ca017939d6ec","Type":"ContainerStarted","Data":"6fc12406f086170f8273384cbc391acb7eefc1110c6355db4b213851afb3c4c6"} Nov 02 22:06:35.409460 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:35.409314 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" event={"ID":"8bf1a756-7259-404a-aa43-ca017939d6ec","Type":"ContainerStarted","Data":"255e3fe4b9ff6e2a1dbc46e4461cf9dc398ef1a6bf8727338d3b61f127c3efc2"} Nov 02 22:06:35.612526 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:35.612470 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" podStartSLOduration=2.61245613 podStartE2EDuration="2.61245613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:06:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:35.563716604 +0000 UTC m=+3488.003291326" watchObservedRunningTime="2026-11-02 22:06:35.61245613 +0000 UTC m=+3488.052030852" Nov 02 22:06:36.193459 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:36.193426 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:36.199612 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:36.199585 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:36.417367 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:36.417335 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:36.418385 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:36.418361 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:06:36.886561 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:36.886525 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:37.208991 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:37.208907 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:37.420211 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:37.420172 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:37.626503 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:37.626434 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:38.903333 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:38.903295 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:39.119856 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:39.119821 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:42.026109 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:42.026069 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:42.236741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:42.236702 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:42.988851 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:42.988775 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:45.100800 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:45.100762 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:47.918473 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:47.918435 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:52.822798 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:52.822758 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:53.035963 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:53.035921 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:53.782932 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:53.782899 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:58.026316 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:06:58.026278 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:58.470254 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:06:58.470159 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:07:02.991715 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:02.991681 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:08.045591 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:08.045552 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:13.137601 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:13.137563 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:17.784625 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:17.784593 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd"] Nov 02 22:07:17.788823 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:17.788795 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd" Nov 02 22:07:17.884767 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:17.884726 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-1538c4-serving-cert\"" Nov 02 22:07:17.887904 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:17.887876 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd"] Nov 02 22:07:17.927226 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:17.927189 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd781594-540f-4aaa-9249-b1a4c20e25ab-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd\" (UID: \"dd781594-540f-4aaa-9249-b1a4c20e25ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd" Nov 02 22:07:17.927427 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:17.927260 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8mhg\" (UniqueName: \"kubernetes.io/projected/dd781594-540f-4aaa-9249-b1a4c20e25ab-kube-api-access-x8mhg\") pod \"maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd\" (UID: \"dd781594-540f-4aaa-9249-b1a4c20e25ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd" Nov 02 22:07:17.927427 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:17.927329 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd781594-540f-4aaa-9249-b1a4c20e25ab-maas-api-tls\") pod \"maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd\" (UID: \"dd781594-540f-4aaa-9249-b1a4c20e25ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd" Nov 02 22:07:17.927427 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:17.927358 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd781594-540f-4aaa-9249-b1a4c20e25ab-metrics-certs\") pod \"maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd\" (UID: \"dd781594-540f-4aaa-9249-b1a4c20e25ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd" Nov 02 22:07:18.028791 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:18.028747 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd781594-540f-4aaa-9249-b1a4c20e25ab-maas-api-tls\") pod \"maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd\" (UID: \"dd781594-540f-4aaa-9249-b1a4c20e25ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd" Nov 02 22:07:18.028977 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:18.028813 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd781594-540f-4aaa-9249-b1a4c20e25ab-metrics-certs\") pod \"maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd\" (UID: \"dd781594-540f-4aaa-9249-b1a4c20e25ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd" Nov 02 22:07:18.028977 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:07:18.028905 4935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-serving-cert: secret "maas-api-e2e-extmod-a-1538c4-serving-cert" not found Nov 02 22:07:18.028977 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:18.028968 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd781594-540f-4aaa-9249-b1a4c20e25ab-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd\" (UID: \"dd781594-540f-4aaa-9249-b1a4c20e25ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd" Nov 02 22:07:18.029103 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:07:18.028990 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd781594-540f-4aaa-9249-b1a4c20e25ab-maas-api-tls podName:dd781594-540f-4aaa-9249-b1a4c20e25ab nodeName:}" failed. No retries permitted until 2026-11-02 22:07:18.528967804 +0000 UTC m=+3530.968542523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dd781594-540f-4aaa-9249-b1a4c20e25ab-maas-api-tls") pod "maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd" (UID: "dd781594-540f-4aaa-9249-b1a4c20e25ab") : secret "maas-api-e2e-extmod-a-1538c4-serving-cert" not found Nov 02 22:07:18.029103 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:18.029062 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8mhg\" (UniqueName: \"kubernetes.io/projected/dd781594-540f-4aaa-9249-b1a4c20e25ab-kube-api-access-x8mhg\") pod \"maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd\" (UID: \"dd781594-540f-4aaa-9249-b1a4c20e25ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd" Nov 02 22:07:18.029747 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:18.029723 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd781594-540f-4aaa-9249-b1a4c20e25ab-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd\" (UID: \"dd781594-540f-4aaa-9249-b1a4c20e25ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd" Nov 02 22:07:18.031503 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:18.031478 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd781594-540f-4aaa-9249-b1a4c20e25ab-metrics-certs\") pod \"maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd\" (UID: \"dd781594-540f-4aaa-9249-b1a4c20e25ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd" Nov 02 22:07:18.055767 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:18.055671 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:18.082890 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:18.082857 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8mhg\" (UniqueName: \"kubernetes.io/projected/dd781594-540f-4aaa-9249-b1a4c20e25ab-kube-api-access-x8mhg\") pod \"maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd\" (UID: \"dd781594-540f-4aaa-9249-b1a4c20e25ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd" Nov 02 22:07:18.269432 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:18.269394 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:18.471639 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:18.471559 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:18.535508 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:18.535463 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd781594-540f-4aaa-9249-b1a4c20e25ab-maas-api-tls\") pod \"maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd\" (UID: \"dd781594-540f-4aaa-9249-b1a4c20e25ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd" Nov 02 22:07:18.537940 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:18.537918 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd781594-540f-4aaa-9249-b1a4c20e25ab-maas-api-tls\") pod \"maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd\" (UID: \"dd781594-540f-4aaa-9249-b1a4c20e25ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd" Nov 02 22:07:18.702762 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:18.702732 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd" Nov 02 22:07:19.072188 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:19.072163 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd"] Nov 02 22:07:19.074787 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:07:19.074745 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd781594_540f_4aaa_9249_b1a4c20e25ab.slice/crio-787e4e33829f5159b64e546e493110a2f72b2df4965827d7bb176eb7116048f9 WatchSource:0}: Error finding container 787e4e33829f5159b64e546e493110a2f72b2df4965827d7bb176eb7116048f9: Status 404 returned error can't find the container with id 787e4e33829f5159b64e546e493110a2f72b2df4965827d7bb176eb7116048f9 Nov 02 22:07:19.750857 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:19.750815 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd" event={"ID":"dd781594-540f-4aaa-9249-b1a4c20e25ab","Type":"ContainerStarted","Data":"787e4e33829f5159b64e546e493110a2f72b2df4965827d7bb176eb7116048f9"} Nov 02 22:07:20.759642 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:20.759590 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd" event={"ID":"dd781594-540f-4aaa-9249-b1a4c20e25ab","Type":"ContainerStarted","Data":"abaafa390049db9401339afd3e7495ffaa099b3f8ac1ad172cd35aa294b6bfc9"} Nov 02 22:07:20.760188 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:20.759663 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd" Nov 02 22:07:20.868543 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:20.868491 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd" podStartSLOduration=2.989116087 podStartE2EDuration="3.86847655s" totalImagesPullingTime="879.360463ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:07:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:07:20.865660473 +0000 UTC m=+3533.305235209" watchObservedRunningTime="2026-11-02 22:07:20.86847655 +0000 UTC m=+3533.308051272" Nov 02 22:07:23.221648 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:23.221606 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:26.774319 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:26.774288 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd" Nov 02 22:07:28.178349 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:28.178301 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:28.472907 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:07:28.472815 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:07:33.297454 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:33.297372 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:35.481442 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:35.481404 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7474-d4bc4a"] Nov 02 22:07:35.485572 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:35.485547 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7474-d4bc4a" Nov 02 22:07:35.546156 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:35.541437 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7474-d4bc4a"] Nov 02 22:07:35.577815 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:35.577788 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5v2nf\"" Nov 02 22:07:35.610659 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:35.610626 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z646\" (UniqueName: \"kubernetes.io/projected/9affdb31-d708-407a-8896-bdfb25e5f134-kube-api-access-2z646\") pod \"test-curl-7474-d4bc4a\" (UID: \"9affdb31-d708-407a-8896-bdfb25e5f134\") " pod="opendatahub/test-curl-7474-d4bc4a" Nov 02 22:07:35.711894 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:35.711854 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z646\" (UniqueName: \"kubernetes.io/projected/9affdb31-d708-407a-8896-bdfb25e5f134-kube-api-access-2z646\") pod \"test-curl-7474-d4bc4a\" (UID: \"9affdb31-d708-407a-8896-bdfb25e5f134\") " pod="opendatahub/test-curl-7474-d4bc4a" Nov 02 22:07:35.764641 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:35.764601 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z646\" (UniqueName: \"kubernetes.io/projected/9affdb31-d708-407a-8896-bdfb25e5f134-kube-api-access-2z646\") pod \"test-curl-7474-d4bc4a\" (UID: \"9affdb31-d708-407a-8896-bdfb25e5f134\") " pod="opendatahub/test-curl-7474-d4bc4a" Nov 02 22:07:35.797810 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:35.797781 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7474-d4bc4a" Nov 02 22:07:36.195066 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:36.195039 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7474-d4bc4a"] Nov 02 22:07:36.196567 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:07:36.196534 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9affdb31_d708_407a_8896_bdfb25e5f134.slice/crio-0fde8907a3009b4360c17f619af23e5bbedb5e9b6555224dd973f9c1f63afa7b WatchSource:0}: Error finding container 0fde8907a3009b4360c17f619af23e5bbedb5e9b6555224dd973f9c1f63afa7b: Status 404 returned error can't find the container with id 0fde8907a3009b4360c17f619af23e5bbedb5e9b6555224dd973f9c1f63afa7b Nov 02 22:07:36.903325 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:36.903280 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7474-d4bc4a" event={"ID":"9affdb31-d708-407a-8896-bdfb25e5f134","Type":"ContainerStarted","Data":"e1fb4dedb895978a19cba3216ae355b2770419ee465969bc46e8e51ca6cbfd08"} Nov 02 22:07:36.903719 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:36.903337 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7474-d4bc4a" event={"ID":"9affdb31-d708-407a-8896-bdfb25e5f134","Type":"ContainerStarted","Data":"0fde8907a3009b4360c17f619af23e5bbedb5e9b6555224dd973f9c1f63afa7b"} Nov 02 22:07:37.009868 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:37.009813 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7474-d4bc4a" podStartSLOduration=2.009797567 podStartE2EDuration="2.009797567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:07:37.007014373 +0000 UTC m=+3549.446589095" watchObservedRunningTime="2026-11-02 22:07:37.009797567 +0000 UTC m=+3549.449372289" Nov 02 22:07:37.208774 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:37.208685 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:37.421711 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:37.421663 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:38.332042 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:38.331952 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:38.453810 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:38.453768 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7474-d4bc4a"] Nov 02 22:07:38.458722 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:38.458685 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7474-d4bc4a"] Nov 02 22:07:38.918245 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:38.918201 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7474-d4bc4a" podUID="9affdb31-d708-407a-8896-bdfb25e5f134" containerName="test-curl-7474-d4bc4a" containerID="cri-o://e1fb4dedb895978a19cba3216ae355b2770419ee465969bc46e8e51ca6cbfd08" gracePeriod=2 Nov 02 22:07:38.966254 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:07:38.966207 4935 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7474-d4bc4a\" is forbidden: User \"system:node:ip-10-0-128-89.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-89.us-west-2.compute.internal' and this object" podUID="9affdb31-d708-407a-8896-bdfb25e5f134" pod="opendatahub/test-curl-7474-d4bc4a" Nov 02 22:07:39.685528 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:39.685165 4935 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": read tcp 10.132.0.2:46282->10.132.0.132:15021: read: connection reset by peer" start-of-body= Nov 02 22:07:39.685528 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:39.685228 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" podUID="d9e7ef62-e7ba-474a-9f7d-650b4c98d0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": read tcp 10.132.0.2:46282->10.132.0.132:15021: read: connection reset by peer" Nov 02 22:07:39.927408 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:39.927330 4935 generic.go:356] "Generic (PLEG): container finished" podID="d9e7ef62-e7ba-474a-9f7d-650b4c98d0de" containerID="aa31aa5acabd4a99eda1990486f703252f925574e10cc265cd0e1840b9735aa2" exitCode=0 Nov 02 22:07:39.927554 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:39.927399 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" event={"ID":"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de","Type":"ContainerDied","Data":"aa31aa5acabd4a99eda1990486f703252f925574e10cc265cd0e1840b9735aa2"} Nov 02 22:07:39.927849 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:39.927831 4935 scope.go:122] "RemoveContainer" containerID="aa31aa5acabd4a99eda1990486f703252f925574e10cc265cd0e1840b9735aa2" Nov 02 22:07:39.929532 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:39.929499 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:07:39.929613 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:39.929567 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:07:39.929613 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:39.929598 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:07:40.976358 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:40.976319 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" event={"ID":"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de","Type":"ContainerStarted","Data":"990cc840cc13c6ae300573492bb9de83a243de0b3d2f8e0f59fc113c5efc5203"} Nov 02 22:07:41.050163 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:41.050079 4935 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 02 22:07:41.050373 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:41.050164 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" podUID="d9e7ef62-e7ba-474a-9f7d-650b4c98d0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 02 22:07:41.050373 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:41.050181 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:07:41.206364 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:41.206337 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7474-d4bc4a" Nov 02 22:07:41.283902 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:41.283868 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9affdb31-d708-407a-8896-bdfb25e5f134-kube-api-access-2z646\" (UniqueName: \"kubernetes.io/projected/9affdb31-d708-407a-8896-bdfb25e5f134-kube-api-access-2z646\") pod \"9affdb31-d708-407a-8896-bdfb25e5f134\" (UID: \"9affdb31-d708-407a-8896-bdfb25e5f134\") " Nov 02 22:07:41.285984 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:41.285948 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9affdb31-d708-407a-8896-bdfb25e5f134-kube-api-access-2z646" pod "9affdb31-d708-407a-8896-bdfb25e5f134" (UID: "9affdb31-d708-407a-8896-bdfb25e5f134"). InnerVolumeSpecName "kube-api-access-2z646". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:41.385776 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:41.385745 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z646\" (UniqueName: \"kubernetes.io/projected/9affdb31-d708-407a-8896-bdfb25e5f134-kube-api-access-2z646\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:07:41.776571 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:41.776538 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9affdb31-d708-407a-8896-bdfb25e5f134" path="/var/lib/kubelet/pods/9affdb31-d708-407a-8896-bdfb25e5f134/volumes" Nov 02 22:07:41.985307 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:41.985272 4935 generic.go:356] "Generic (PLEG): container finished" podID="9affdb31-d708-407a-8896-bdfb25e5f134" containerID="e1fb4dedb895978a19cba3216ae355b2770419ee465969bc46e8e51ca6cbfd08" exitCode=137 Nov 02 22:07:41.985803 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:41.985327 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7474-d4bc4a" Nov 02 22:07:41.985803 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:41.985364 4935 scope.go:122] "RemoveContainer" containerID="e1fb4dedb895978a19cba3216ae355b2770419ee465969bc46e8e51ca6cbfd08" Nov 02 22:07:41.995007 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:41.994989 4935 scope.go:122] "RemoveContainer" containerID="e1fb4dedb895978a19cba3216ae355b2770419ee465969bc46e8e51ca6cbfd08" Nov 02 22:07:41.995292 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:07:41.995274 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1fb4dedb895978a19cba3216ae355b2770419ee465969bc46e8e51ca6cbfd08\": container with ID starting with e1fb4dedb895978a19cba3216ae355b2770419ee465969bc46e8e51ca6cbfd08 not found: ID does not exist" containerID="e1fb4dedb895978a19cba3216ae355b2770419ee465969bc46e8e51ca6cbfd08" Nov 02 22:07:41.995344 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:41.995300 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1fb4dedb895978a19cba3216ae355b2770419ee465969bc46e8e51ca6cbfd08"} err="failed to get container status \"e1fb4dedb895978a19cba3216ae355b2770419ee465969bc46e8e51ca6cbfd08\": rpc error: code = NotFound desc = could not find container \"e1fb4dedb895978a19cba3216ae355b2770419ee465969bc46e8e51ca6cbfd08\": container with ID starting with e1fb4dedb895978a19cba3216ae355b2770419ee465969bc46e8e51ca6cbfd08 not found: ID does not exist" Nov 02 22:07:42.039608 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:42.039517 4935 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 02 22:07:42.039852 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:42.039821 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" podUID="d9e7ef62-e7ba-474a-9f7d-650b4c98d0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 02 22:07:43.038603 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:43.038561 4935 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 02 22:07:43.039247 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:43.039211 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" podUID="d9e7ef62-e7ba-474a-9f7d-650b4c98d0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 02 22:07:43.477046 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:43.476953 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:44.041155 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:44.039564 4935 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 02 22:07:44.041155 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:44.039633 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" podUID="d9e7ef62-e7ba-474a-9f7d-650b4c98d0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 02 22:07:45.042157 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:45.041533 4935 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 02 22:07:45.042157 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:45.041612 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" podUID="d9e7ef62-e7ba-474a-9f7d-650b4c98d0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 02 22:07:46.038571 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:46.038531 4935 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 02 22:07:46.038797 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:46.038599 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" podUID="d9e7ef62-e7ba-474a-9f7d-650b4c98d0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 02 22:07:47.043605 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:47.043573 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:07:47.044113 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:47.043982 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:07:47.044831 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:47.044807 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:07:48.522188 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:48.522140 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:53.505664 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:53.505626 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:58.471229 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:07:58.471173 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:07:58.520907 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:07:58.520866 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:03.400496 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:03.400445 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:05.744753 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:05.744673 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:08.450942 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:08.450890 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:13.103096 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:13.103061 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj"] Nov 02 22:08:13.103622 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:13.103605 4935 cpu_manager.go:414] "containerMap: removing container" podUID="9affdb31-d708-407a-8896-bdfb25e5f134" containerName="test-curl-7474-d4bc4a" Nov 02 22:08:13.103666 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:13.103624 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9affdb31-d708-407a-8896-bdfb25e5f134" containerName="test-curl-7474-d4bc4a" Nov 02 22:08:13.103750 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:13.103740 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="9affdb31-d708-407a-8896-bdfb25e5f134" containerName="test-curl-7474-d4bc4a" Nov 02 22:08:13.106238 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:13.106217 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj" Nov 02 22:08:13.192562 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:13.192529 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e33d3d-0533-44c0-842e-c5122d9a3dae-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj\" (UID: \"67e33d3d-0533-44c0-842e-c5122d9a3dae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj" Nov 02 22:08:13.192728 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:13.192583 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67e33d3d-0533-44c0-842e-c5122d9a3dae-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj\" (UID: \"67e33d3d-0533-44c0-842e-c5122d9a3dae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj" Nov 02 22:08:13.192728 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:13.192654 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67e33d3d-0533-44c0-842e-c5122d9a3dae-metrics-certs\") pod \"maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj\" (UID: \"67e33d3d-0533-44c0-842e-c5122d9a3dae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj" Nov 02 22:08:13.192728 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:13.192694 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zbnf\" (UniqueName: \"kubernetes.io/projected/67e33d3d-0533-44c0-842e-c5122d9a3dae-kube-api-access-8zbnf\") pod \"maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj\" (UID: \"67e33d3d-0533-44c0-842e-c5122d9a3dae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj" Nov 02 22:08:13.202632 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:13.202612 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-6ba06f-serving-cert\"" Nov 02 22:08:13.205704 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:13.205682 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj"] Nov 02 22:08:13.293657 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:13.293624 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e33d3d-0533-44c0-842e-c5122d9a3dae-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj\" (UID: \"67e33d3d-0533-44c0-842e-c5122d9a3dae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj" Nov 02 22:08:13.293862 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:13.293682 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67e33d3d-0533-44c0-842e-c5122d9a3dae-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj\" (UID: \"67e33d3d-0533-44c0-842e-c5122d9a3dae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj" Nov 02 22:08:13.293862 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:13.293716 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67e33d3d-0533-44c0-842e-c5122d9a3dae-metrics-certs\") pod \"maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj\" (UID: \"67e33d3d-0533-44c0-842e-c5122d9a3dae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj" Nov 02 22:08:13.293862 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:13.293736 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zbnf\" (UniqueName: \"kubernetes.io/projected/67e33d3d-0533-44c0-842e-c5122d9a3dae-kube-api-access-8zbnf\") pod \"maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj\" (UID: \"67e33d3d-0533-44c0-842e-c5122d9a3dae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj" Nov 02 22:08:13.293862 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:08:13.293856 4935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-serving-cert: secret "maas-api-e2e-extmod-b-6ba06f-serving-cert" not found Nov 02 22:08:13.294101 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:08:13.293938 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67e33d3d-0533-44c0-842e-c5122d9a3dae-maas-api-tls podName:67e33d3d-0533-44c0-842e-c5122d9a3dae nodeName:}" failed. No retries permitted until 2026-11-02 22:08:13.793916078 +0000 UTC m=+3586.233490799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/67e33d3d-0533-44c0-842e-c5122d9a3dae-maas-api-tls") pod "maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj" (UID: "67e33d3d-0533-44c0-842e-c5122d9a3dae") : secret "maas-api-e2e-extmod-b-6ba06f-serving-cert" not found Nov 02 22:08:13.294428 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:13.294403 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e33d3d-0533-44c0-842e-c5122d9a3dae-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj\" (UID: \"67e33d3d-0533-44c0-842e-c5122d9a3dae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj" Nov 02 22:08:13.296284 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:13.296262 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67e33d3d-0533-44c0-842e-c5122d9a3dae-metrics-certs\") pod \"maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj\" (UID: \"67e33d3d-0533-44c0-842e-c5122d9a3dae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj" Nov 02 22:08:13.347364 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:13.347334 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zbnf\" (UniqueName: \"kubernetes.io/projected/67e33d3d-0533-44c0-842e-c5122d9a3dae-kube-api-access-8zbnf\") pod \"maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj\" (UID: \"67e33d3d-0533-44c0-842e-c5122d9a3dae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj" Nov 02 22:08:13.377430 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:13.377357 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:13.574040 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:13.573995 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:13.799399 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:13.799357 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67e33d3d-0533-44c0-842e-c5122d9a3dae-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj\" (UID: \"67e33d3d-0533-44c0-842e-c5122d9a3dae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj" Nov 02 22:08:13.801803 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:13.801777 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67e33d3d-0533-44c0-842e-c5122d9a3dae-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj\" (UID: \"67e33d3d-0533-44c0-842e-c5122d9a3dae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj" Nov 02 22:08:14.019270 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:14.019232 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj" Nov 02 22:08:14.391789 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:14.391759 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj"] Nov 02 22:08:14.393162 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:08:14.393109 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e33d3d_0533_44c0_842e_c5122d9a3dae.slice/crio-db54041dacdbc7bf7b0267ed6b36d1c6894e6d31d9a5d7d7c61c1912ba4b4dc4 WatchSource:0}: Error finding container db54041dacdbc7bf7b0267ed6b36d1c6894e6d31d9a5d7d7c61c1912ba4b4dc4: Status 404 returned error can't find the container with id db54041dacdbc7bf7b0267ed6b36d1c6894e6d31d9a5d7d7c61c1912ba4b4dc4 Nov 02 22:08:15.204287 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:15.204248 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj" event={"ID":"67e33d3d-0533-44c0-842e-c5122d9a3dae","Type":"ContainerStarted","Data":"db54041dacdbc7bf7b0267ed6b36d1c6894e6d31d9a5d7d7c61c1912ba4b4dc4"} Nov 02 22:08:16.213718 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:16.213681 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj" event={"ID":"67e33d3d-0533-44c0-842e-c5122d9a3dae","Type":"ContainerStarted","Data":"af5cf64ccfb51f90c6ec3b5123806b1f69c0294788b8e60c1ece22dc9f1e8eb8"} Nov 02 22:08:16.214267 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:16.213890 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj" Nov 02 22:08:16.322231 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:16.322168 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj" podStartSLOduration=2.35138658 podStartE2EDuration="3.322148701s" totalImagesPullingTime="970.762121ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:16.320876432 +0000 UTC m=+3588.760451155" watchObservedRunningTime="2026-11-02 22:08:16.322148701 +0000 UTC m=+3588.761723423" Nov 02 22:08:16.707967 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:16.707932 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:18.613573 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:18.613538 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:22.227355 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:22.227322 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj" Nov 02 22:08:22.354688 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:22.354651 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:23.634462 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:23.634420 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:23.837582 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:23.837542 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:24.841840 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:24.841797 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:27.954382 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:27.954336 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:28.453681 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:28.453649 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 22:08:28.464600 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:28.464565 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 22:08:28.475728 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:08:28.475690 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:08:28.479340 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:28.479320 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 22:08:28.488185 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:28.488164 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 22:08:28.720936 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:28.720826 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:30.695690 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:30.695098 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:30.914479 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:30.914433 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:34.031407 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:34.031365 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:34.617278 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:34.617239 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:35.502311 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:35.502277 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:35.848840 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:35.848807 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:36.581718 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:36.581682 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:37.534622 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:37.534585 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:39.547105 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:39.547067 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:39.767051 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:39.767017 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:40.020101 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:40.020059 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:40.249394 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:40.249358 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:45.065249 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:45.065212 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:46.606303 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:46.606264 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:46.833842 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:46.833799 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:48.949646 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:48.949610 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:49.958355 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:49.958309 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:54.417807 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:54.417766 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:54.621062 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:54.621010 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:54.838747 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:54.838638 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:55.163400 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:55.163250 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:58.481808 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:08:58.481754 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:08:59.481038 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:59.480995 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd"] Nov 02 22:08:59.481355 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:59.481224 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd" podUID="dd781594-540f-4aaa-9249-b1a4c20e25ab" containerName="maas-api" containerID="cri-o://abaafa390049db9401339afd3e7495ffaa099b3f8ac1ad172cd35aa294b6bfc9" gracePeriod=30 Nov 02 22:08:59.680454 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:59.680347 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:59.938151 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:59.934293 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd" Nov 02 22:08:59.978579 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:59.978531 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd781594-540f-4aaa-9249-b1a4c20e25ab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd781594-540f-4aaa-9249-b1a4c20e25ab-metrics-certs\") pod \"dd781594-540f-4aaa-9249-b1a4c20e25ab\" (UID: \"dd781594-540f-4aaa-9249-b1a4c20e25ab\") " Nov 02 22:08:59.978779 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:59.978619 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd781594-540f-4aaa-9249-b1a4c20e25ab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd781594-540f-4aaa-9249-b1a4c20e25ab-maas-api-tls\") pod \"dd781594-540f-4aaa-9249-b1a4c20e25ab\" (UID: \"dd781594-540f-4aaa-9249-b1a4c20e25ab\") " Nov 02 22:08:59.978779 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:59.978718 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd781594-540f-4aaa-9249-b1a4c20e25ab-kube-api-access-x8mhg\" (UniqueName: \"kubernetes.io/projected/dd781594-540f-4aaa-9249-b1a4c20e25ab-kube-api-access-x8mhg\") pod \"dd781594-540f-4aaa-9249-b1a4c20e25ab\" (UID: \"dd781594-540f-4aaa-9249-b1a4c20e25ab\") " Nov 02 22:08:59.978903 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:59.978783 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd781594-540f-4aaa-9249-b1a4c20e25ab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd781594-540f-4aaa-9249-b1a4c20e25ab-openshift-service-ca\") pod \"dd781594-540f-4aaa-9249-b1a4c20e25ab\" (UID: \"dd781594-540f-4aaa-9249-b1a4c20e25ab\") " Nov 02 22:08:59.979860 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:59.979829 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd781594-540f-4aaa-9249-b1a4c20e25ab-openshift-service-ca" pod "dd781594-540f-4aaa-9249-b1a4c20e25ab" (UID: "dd781594-540f-4aaa-9249-b1a4c20e25ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:08:59.981928 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:59.981902 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd781594-540f-4aaa-9249-b1a4c20e25ab-metrics-certs" pod "dd781594-540f-4aaa-9249-b1a4c20e25ab" (UID: "dd781594-540f-4aaa-9249-b1a4c20e25ab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:08:59.982014 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:59.981937 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd781594-540f-4aaa-9249-b1a4c20e25ab-kube-api-access-x8mhg" pod "dd781594-540f-4aaa-9249-b1a4c20e25ab" (UID: "dd781594-540f-4aaa-9249-b1a4c20e25ab"). InnerVolumeSpecName "kube-api-access-x8mhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:08:59.983034 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:08:59.982540 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd781594-540f-4aaa-9249-b1a4c20e25ab-maas-api-tls" pod "dd781594-540f-4aaa-9249-b1a4c20e25ab" (UID: "dd781594-540f-4aaa-9249-b1a4c20e25ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:09:00.080563 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:00.080521 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd781594-540f-4aaa-9249-b1a4c20e25ab-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:00.080563 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:00.080562 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8mhg\" (UniqueName: \"kubernetes.io/projected/dd781594-540f-4aaa-9249-b1a4c20e25ab-kube-api-access-x8mhg\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:00.080823 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:00.080579 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd781594-540f-4aaa-9249-b1a4c20e25ab-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:00.080823 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:00.080595 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd781594-540f-4aaa-9249-b1a4c20e25ab-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:00.516403 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:00.516368 4935 generic.go:356] "Generic (PLEG): container finished" podID="dd781594-540f-4aaa-9249-b1a4c20e25ab" containerID="abaafa390049db9401339afd3e7495ffaa099b3f8ac1ad172cd35aa294b6bfc9" exitCode=0 Nov 02 22:09:00.516598 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:00.516433 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd" Nov 02 22:09:00.516598 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:00.516460 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd" event={"ID":"dd781594-540f-4aaa-9249-b1a4c20e25ab","Type":"ContainerDied","Data":"abaafa390049db9401339afd3e7495ffaa099b3f8ac1ad172cd35aa294b6bfc9"} Nov 02 22:09:00.516598 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:00.516530 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd" event={"ID":"dd781594-540f-4aaa-9249-b1a4c20e25ab","Type":"ContainerDied","Data":"787e4e33829f5159b64e546e493110a2f72b2df4965827d7bb176eb7116048f9"} Nov 02 22:09:00.516598 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:00.516559 4935 scope.go:122] "RemoveContainer" containerID="abaafa390049db9401339afd3e7495ffaa099b3f8ac1ad172cd35aa294b6bfc9" Nov 02 22:09:00.529031 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:00.529004 4935 scope.go:122] "RemoveContainer" containerID="abaafa390049db9401339afd3e7495ffaa099b3f8ac1ad172cd35aa294b6bfc9" Nov 02 22:09:00.529349 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:09:00.529321 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abaafa390049db9401339afd3e7495ffaa099b3f8ac1ad172cd35aa294b6bfc9\": container with ID starting with abaafa390049db9401339afd3e7495ffaa099b3f8ac1ad172cd35aa294b6bfc9 not found: ID does not exist" containerID="abaafa390049db9401339afd3e7495ffaa099b3f8ac1ad172cd35aa294b6bfc9" Nov 02 22:09:00.529441 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:00.529364 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abaafa390049db9401339afd3e7495ffaa099b3f8ac1ad172cd35aa294b6bfc9"} err="failed to get container status \"abaafa390049db9401339afd3e7495ffaa099b3f8ac1ad172cd35aa294b6bfc9\": rpc error: code = NotFound desc = could not find container \"abaafa390049db9401339afd3e7495ffaa099b3f8ac1ad172cd35aa294b6bfc9\": container with ID starting with abaafa390049db9401339afd3e7495ffaa099b3f8ac1ad172cd35aa294b6bfc9 not found: ID does not exist" Nov 02 22:09:00.675689 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:00.675648 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd"] Nov 02 22:09:00.721024 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:00.720979 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1538c4-698689f77b-v5tqd"] Nov 02 22:09:00.781507 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:00.781427 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:01.553753 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:01.553714 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:01.777868 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:01.777834 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd781594-540f-4aaa-9249-b1a4c20e25ab" path="/var/lib/kubelet/pods/dd781594-540f-4aaa-9249-b1a4c20e25ab/volumes" Nov 02 22:09:02.206100 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:02.206059 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:03.553468 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:03.553428 4935 generic.go:356] "Generic (PLEG): container finished" podID="8bf1a756-7259-404a-aa43-ca017939d6ec" containerID="6fc12406f086170f8273384cbc391acb7eefc1110c6355db4b213851afb3c4c6" exitCode=0 Nov 02 22:09:03.553913 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:03.553506 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" event={"ID":"8bf1a756-7259-404a-aa43-ca017939d6ec","Type":"ContainerDied","Data":"6fc12406f086170f8273384cbc391acb7eefc1110c6355db4b213851afb3c4c6"} Nov 02 22:09:03.554031 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:03.554012 4935 scope.go:122] "RemoveContainer" containerID="6fc12406f086170f8273384cbc391acb7eefc1110c6355db4b213851afb3c4c6" Nov 02 22:09:03.555840 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:03.555809 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:09:03.555961 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:03.555896 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:09:03.555961 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:03.555946 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:09:04.562554 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:04.562513 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" event={"ID":"8bf1a756-7259-404a-aa43-ca017939d6ec","Type":"ContainerStarted","Data":"a91e1b8e72446adc8ab4844f64d594582d0cdad4d3324d5a8c5a54b3ed2ba821"} Nov 02 22:09:05.193012 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:05.192979 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:09:05.194465 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:05.194435 4935 patch_prober.go:29] interesting pod/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 22:09:05.194609 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:05.194494 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" podUID="8bf1a756-7259-404a-aa43-ca017939d6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 22:09:06.193873 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:06.193831 4935 patch_prober.go:29] interesting pod/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 22:09:06.194403 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:06.193894 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" podUID="8bf1a756-7259-404a-aa43-ca017939d6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 22:09:06.287267 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:06.287215 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:06.417432 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:06.417399 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:09:06.502080 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:06.501986 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:06.693463 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:06.693414 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:07.194053 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:07.194019 4935 patch_prober.go:29] interesting pod/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 22:09:07.194491 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:07.194082 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" podUID="8bf1a756-7259-404a-aa43-ca017939d6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 22:09:07.846151 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:07.845958 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:08.193310 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:08.193195 4935 patch_prober.go:29] interesting pod/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 22:09:08.193310 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:08.193256 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" podUID="8bf1a756-7259-404a-aa43-ca017939d6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 22:09:08.335908 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:08.335829 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:09.193696 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:09.193658 4935 patch_prober.go:29] interesting pod/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 22:09:09.193875 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:09.193713 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" podUID="8bf1a756-7259-404a-aa43-ca017939d6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 22:09:10.303304 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:10.303272 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:09:10.305007 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:10.304985 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:09:11.744151 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:11.739655 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:12.067942 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:12.067888 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:12.287164 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:12.287096 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:15.504078 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:15.504040 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:17.320840 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:17.320783 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:22.214543 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:22.214496 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:27.173487 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:27.173444 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:28.474169 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:09:28.474097 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:09:32.201667 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:32.201615 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:37.149456 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:37.149422 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:37.882455 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:37.882407 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:42.107616 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:42.107582 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:42.325207 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:42.325163 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:47.263536 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:47.263491 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:48.630893 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:48.630855 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:49.732945 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:49.732912 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:49.951322 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:49.951286 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:52.767252 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:52.767215 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:54.432420 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:54.432385 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:57.976488 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:09:57.976447 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:58.469862 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:09:58.469822 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:10:03.526916 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:03.526871 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:08.487771 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:08.487735 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:08.705116 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:08.705081 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:08.945489 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:08.945455 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:10.717213 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:10.717172 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:10.989585 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:10.989508 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:13.072648 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:13.072618 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:13.300723 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:13.300694 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:13.654681 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:13.654616 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:16.481475 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:16.481435 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:18.662883 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:18.662843 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:19.218352 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:19.218320 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:19.438069 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:19.438023 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:20.303559 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:20.303520 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:23.641419 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:23.641359 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:25.368727 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:25.368661 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:25.585991 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:25.585953 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:27.111335 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:27.111299 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:28.469359 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:10:28.469314 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:10:28.754591 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:28.754501 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:31.500807 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.500769 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m"] Nov 02 22:10:31.501355 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.501338 4935 cpu_manager.go:414] "containerMap: removing container" podUID="dd781594-540f-4aaa-9249-b1a4c20e25ab" containerName="maas-api" Nov 02 22:10:31.501408 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.501355 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd781594-540f-4aaa-9249-b1a4c20e25ab" containerName="maas-api" Nov 02 22:10:31.501473 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.501463 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd781594-540f-4aaa-9249-b1a4c20e25ab" containerName="maas-api" Nov 02 22:10:31.504935 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.504916 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:31.661669 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.661620 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b9q8m\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:31.661872 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.661711 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b9q8m\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:31.661872 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.661784 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b9q8m\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:31.661872 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.661853 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b9q8m\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:31.662051 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.661897 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b9q8m\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:31.662051 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.661926 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn4jz\" (UniqueName: \"kubernetes.io/projected/2750bd81-031e-4b2c-bb07-a0ab009da8aa-kube-api-access-wn4jz\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b9q8m\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:31.662051 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.661983 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b9q8m\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:31.662051 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.662046 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b9q8m\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:31.662302 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.662084 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b9q8m\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:31.694255 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.694226 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m"] Nov 02 22:10:31.763189 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.763152 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b9q8m\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:31.763372 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.763318 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b9q8m\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:31.763451 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.763382 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b9q8m\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:31.763451 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.763418 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn4jz\" (UniqueName: \"kubernetes.io/projected/2750bd81-031e-4b2c-bb07-a0ab009da8aa-kube-api-access-wn4jz\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b9q8m\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:31.763561 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.763482 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b9q8m\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:31.763561 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.763547 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b9q8m\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:31.763665 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.763590 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b9q8m\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:31.763665 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.763626 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b9q8m\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:31.763665 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.763641 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b9q8m\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:31.763826 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.763680 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b9q8m\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:31.763826 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.763735 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b9q8m\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:31.763966 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.763943 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b9q8m\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:31.764121 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.764097 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b9q8m\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:31.765277 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.765243 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b9q8m\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:31.766189 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.766163 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b9q8m\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:31.766451 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.766433 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b9q8m\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:31.822546 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.822460 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b9q8m\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:31.832335 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.823828 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn4jz\" (UniqueName: \"kubernetes.io/projected/2750bd81-031e-4b2c-bb07-a0ab009da8aa-kube-api-access-wn4jz\") pod \"maas-default-gateway-openshift-default-fcb75c45f-b9q8m\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:31.979616 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:31.979555 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:32.118799 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:32.118711 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:32.202698 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:32.202664 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:32.545085 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:32.545055 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m"] Nov 02 22:10:32.547733 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:10:32.547697 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2750bd81_031e_4b2c_bb07_a0ab009da8aa.slice/crio-b6e7b88a65730057382fd1b34629b3cc5fcb69498d73aef2d387f48c0eb9d668 WatchSource:0}: Error finding container b6e7b88a65730057382fd1b34629b3cc5fcb69498d73aef2d387f48c0eb9d668: Status 404 returned error can't find the container with id b6e7b88a65730057382fd1b34629b3cc5fcb69498d73aef2d387f48c0eb9d668 Nov 02 22:10:32.551000 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:32.550959 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:10:32.551090 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:32.551053 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:10:32.551159 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:32.551093 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:10:33.215912 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:33.215859 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" event={"ID":"2750bd81-031e-4b2c-bb07-a0ab009da8aa","Type":"ContainerStarted","Data":"be4bc50f94ecdc015b399593f4676d1d52f928eee472020cdd545e1e95bfd0d8"} Nov 02 22:10:33.216111 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:33.215963 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" event={"ID":"2750bd81-031e-4b2c-bb07-a0ab009da8aa","Type":"ContainerStarted","Data":"b6e7b88a65730057382fd1b34629b3cc5fcb69498d73aef2d387f48c0eb9d668"} Nov 02 22:10:33.421123 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:33.421061 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" podStartSLOduration=2.421046041 podStartE2EDuration="2.421046041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:33.372188313 +0000 UTC m=+3725.811763036" watchObservedRunningTime="2026-11-02 22:10:33.421046041 +0000 UTC m=+3725.860620763" Nov 02 22:10:33.867180 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:33.867109 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:34.085651 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:34.085497 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:34.119509 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:34.119428 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:34.121213 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:34.121180 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-b9q8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 02 22:10:34.121417 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:34.121241 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" podUID="2750bd81-031e-4b2c-bb07-a0ab009da8aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 02 22:10:34.298878 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:34.298828 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:35.119565 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:35.119526 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-b9q8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 02 22:10:35.119965 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:35.119584 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" podUID="2750bd81-031e-4b2c-bb07-a0ab009da8aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 02 22:10:36.119978 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:36.119939 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-b9q8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 02 22:10:36.120419 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:36.120019 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" podUID="2750bd81-031e-4b2c-bb07-a0ab009da8aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 02 22:10:37.119195 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:37.119161 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-b9q8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 02 22:10:37.119384 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:37.119225 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" podUID="2750bd81-031e-4b2c-bb07-a0ab009da8aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 02 22:10:38.119332 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:38.119251 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-b9q8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 02 22:10:38.119332 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:38.119308 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" podUID="2750bd81-031e-4b2c-bb07-a0ab009da8aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 02 22:10:38.257788 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:38.257732 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:38.491746 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:38.491653 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:39.048546 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:39.048496 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:39.120074 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:39.120034 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-b9q8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 02 22:10:39.120521 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:39.120116 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" podUID="2750bd81-031e-4b2c-bb07-a0ab009da8aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 02 22:10:40.119754 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:40.119713 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-b9q8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 02 22:10:40.119949 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:40.119786 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" podUID="2750bd81-031e-4b2c-bb07-a0ab009da8aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 02 22:10:40.461639 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:40.461548 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:40.689774 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:40.689734 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:41.493479 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:41.493443 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:41.494786 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:41.494760 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:41.495510 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:41.495489 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:10:42.341611 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:42.341567 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:43.832676 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:43.832641 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:44.242596 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:44.242496 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:44.709995 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:44.709963 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:44.928535 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:44.928496 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:46.597987 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.597953 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k"] Nov 02 22:10:46.602988 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.602965 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:46.701564 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.701529 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k"] Nov 02 22:10:46.728546 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.728513 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-f7c8k\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:46.728546 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.728548 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-f7c8k\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:46.728797 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.728593 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-f7c8k\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:46.728797 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.728698 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-f7c8k\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:46.728797 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.728730 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-f7c8k\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:46.728970 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.728813 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-f7c8k\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:46.728970 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.728839 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78qbp\" (UniqueName: \"kubernetes.io/projected/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-kube-api-access-78qbp\") pod \"maas-default-gateway-openshift-default-fcb75c45f-f7c8k\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:46.728970 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.728865 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-f7c8k\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:46.728970 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.728897 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-f7c8k\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:46.829665 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.829627 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-f7c8k\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:46.829863 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.829694 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78qbp\" (UniqueName: \"kubernetes.io/projected/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-kube-api-access-78qbp\") pod \"maas-default-gateway-openshift-default-fcb75c45f-f7c8k\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:46.829863 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.829726 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-f7c8k\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:46.829863 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.829767 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-f7c8k\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:46.829863 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.829812 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-f7c8k\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:46.829863 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.829840 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-f7c8k\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:46.830216 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.829891 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-f7c8k\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:46.830216 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.829969 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-f7c8k\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:46.830216 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.830007 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-f7c8k\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:46.830216 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.830207 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-f7c8k\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:46.830430 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.830398 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-f7c8k\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:46.830620 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.830595 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-f7c8k\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:46.830715 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.830669 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-f7c8k\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:46.830869 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.830845 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-f7c8k\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:46.832306 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.832277 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-f7c8k\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:46.832502 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.832480 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-f7c8k\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:46.882393 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.882328 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-f7c8k\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:46.882393 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.882341 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78qbp\" (UniqueName: \"kubernetes.io/projected/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-kube-api-access-78qbp\") pod \"maas-default-gateway-openshift-default-fcb75c45f-f7c8k\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:46.920102 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:46.920079 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:47.350471 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:47.350437 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k"] Nov 02 22:10:47.351194 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:10:47.351158 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232da504_01ae_4fb4_86cb_43c8ca0f6a7a.slice/crio-f46a817823e2e239ecc83108c8ac37ca84988deead3c63aff533550d06afc483 WatchSource:0}: Error finding container f46a817823e2e239ecc83108c8ac37ca84988deead3c63aff533550d06afc483: Status 404 returned error can't find the container with id f46a817823e2e239ecc83108c8ac37ca84988deead3c63aff533550d06afc483 Nov 02 22:10:47.354258 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:47.354218 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:10:47.354393 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:47.354303 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:10:47.354393 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:47.354349 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:10:48.363818 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:48.363770 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" event={"ID":"232da504-01ae-4fb4-86cb-43c8ca0f6a7a","Type":"ContainerStarted","Data":"8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5"} Nov 02 22:10:48.364281 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:48.363839 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" event={"ID":"232da504-01ae-4fb4-86cb-43c8ca0f6a7a","Type":"ContainerStarted","Data":"f46a817823e2e239ecc83108c8ac37ca84988deead3c63aff533550d06afc483"} Nov 02 22:10:48.520561 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:48.520506 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" podStartSLOduration=2.520491498 podStartE2EDuration="2.520491498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:48.517343315 +0000 UTC m=+3740.956918037" watchObservedRunningTime="2026-11-02 22:10:48.520491498 +0000 UTC m=+3740.960066218" Nov 02 22:10:48.920907 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:48.920862 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:48.922821 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:48.922788 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-f7c8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Nov 02 22:10:48.922966 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:48.922852 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" podUID="232da504-01ae-4fb4-86cb-43c8ca0f6a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Nov 02 22:10:49.071532 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:49.071482 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:49.319365 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:49.319327 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:49.551532 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:49.551496 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:49.920844 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:49.920812 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-f7c8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Nov 02 22:10:49.921107 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:49.920862 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" podUID="232da504-01ae-4fb4-86cb-43c8ca0f6a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Nov 02 22:10:50.921304 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:50.921256 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-f7c8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Nov 02 22:10:50.921699 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:50.921324 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" podUID="232da504-01ae-4fb4-86cb-43c8ca0f6a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Nov 02 22:10:51.921313 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:51.921280 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-f7c8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Nov 02 22:10:51.921713 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:51.921347 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" podUID="232da504-01ae-4fb4-86cb-43c8ca0f6a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Nov 02 22:10:52.921312 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:52.921282 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-f7c8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Nov 02 22:10:52.921766 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:52.921341 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" podUID="232da504-01ae-4fb4-86cb-43c8ca0f6a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Nov 02 22:10:52.941782 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:52.941746 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:53.184523 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:53.184392 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:53.920649 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:53.920600 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-f7c8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Nov 02 22:10:53.920862 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:53.920656 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" podUID="232da504-01ae-4fb4-86cb-43c8ca0f6a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Nov 02 22:10:54.920431 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:54.920395 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-f7c8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Nov 02 22:10:54.920937 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:54.920460 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" podUID="232da504-01ae-4fb4-86cb-43c8ca0f6a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Nov 02 22:10:55.188043 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:55.187953 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:56.256001 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:56.255970 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:56.256539 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:56.256462 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:56.257363 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:56.257344 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:10:58.470604 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:10:58.470561 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:10:59.094501 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:59.094461 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:59.305589 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:10:59.305556 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:00.819768 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:00.819710 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:07.171620 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:07.171580 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:10.655999 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:10.655959 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7"] Nov 02 22:11:10.656457 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:10.656187 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7" podUID="ec842943-f0f8-4fd0-b32d-96314b30077f" containerName="maas-api" containerID="cri-o://d480510702914ea1cd9e9fb1b8cc72c1de5ba89a5f3c2d6fc08021faec521427" gracePeriod=30 Nov 02 22:11:10.859454 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:10.859421 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:10.954251 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:10.954219 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7" Nov 02 22:11:10.977354 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:10.977314 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec842943-f0f8-4fd0-b32d-96314b30077f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec842943-f0f8-4fd0-b32d-96314b30077f-maas-api-tls\") pod \"ec842943-f0f8-4fd0-b32d-96314b30077f\" (UID: \"ec842943-f0f8-4fd0-b32d-96314b30077f\") " Nov 02 22:11:10.977542 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:10.977412 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec842943-f0f8-4fd0-b32d-96314b30077f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec842943-f0f8-4fd0-b32d-96314b30077f-metrics-certs\") pod \"ec842943-f0f8-4fd0-b32d-96314b30077f\" (UID: \"ec842943-f0f8-4fd0-b32d-96314b30077f\") " Nov 02 22:11:10.977542 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:10.977459 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec842943-f0f8-4fd0-b32d-96314b30077f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec842943-f0f8-4fd0-b32d-96314b30077f-openshift-service-ca\") pod \"ec842943-f0f8-4fd0-b32d-96314b30077f\" (UID: \"ec842943-f0f8-4fd0-b32d-96314b30077f\") " Nov 02 22:11:10.977670 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:10.977548 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec842943-f0f8-4fd0-b32d-96314b30077f-kube-api-access-tkk9c\" (UniqueName: \"kubernetes.io/projected/ec842943-f0f8-4fd0-b32d-96314b30077f-kube-api-access-tkk9c\") pod \"ec842943-f0f8-4fd0-b32d-96314b30077f\" (UID: \"ec842943-f0f8-4fd0-b32d-96314b30077f\") " Nov 02 22:11:10.978063 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:10.977893 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec842943-f0f8-4fd0-b32d-96314b30077f-openshift-service-ca" pod "ec842943-f0f8-4fd0-b32d-96314b30077f" (UID: "ec842943-f0f8-4fd0-b32d-96314b30077f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:11:10.978209 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:10.978100 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec842943-f0f8-4fd0-b32d-96314b30077f-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:10.979844 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:10.979818 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec842943-f0f8-4fd0-b32d-96314b30077f-kube-api-access-tkk9c" pod "ec842943-f0f8-4fd0-b32d-96314b30077f" (UID: "ec842943-f0f8-4fd0-b32d-96314b30077f"). InnerVolumeSpecName "kube-api-access-tkk9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:11:10.980125 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:10.980099 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec842943-f0f8-4fd0-b32d-96314b30077f-metrics-certs" pod "ec842943-f0f8-4fd0-b32d-96314b30077f" (UID: "ec842943-f0f8-4fd0-b32d-96314b30077f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:11:10.980325 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:10.980306 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec842943-f0f8-4fd0-b32d-96314b30077f-maas-api-tls" pod "ec842943-f0f8-4fd0-b32d-96314b30077f" (UID: "ec842943-f0f8-4fd0-b32d-96314b30077f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:11:11.080947 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:11.080909 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec842943-f0f8-4fd0-b32d-96314b30077f-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:11.081236 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:11.081215 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkk9c\" (UniqueName: \"kubernetes.io/projected/ec842943-f0f8-4fd0-b32d-96314b30077f-kube-api-access-tkk9c\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:11.081356 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:11.081343 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec842943-f0f8-4fd0-b32d-96314b30077f-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:11.534483 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:11.534449 4935 generic.go:356] "Generic (PLEG): container finished" podID="ec842943-f0f8-4fd0-b32d-96314b30077f" containerID="d480510702914ea1cd9e9fb1b8cc72c1de5ba89a5f3c2d6fc08021faec521427" exitCode=0 Nov 02 22:11:11.534483 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:11.534486 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7" Nov 02 22:11:11.534738 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:11.534526 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7" event={"ID":"ec842943-f0f8-4fd0-b32d-96314b30077f","Type":"ContainerDied","Data":"d480510702914ea1cd9e9fb1b8cc72c1de5ba89a5f3c2d6fc08021faec521427"} Nov 02 22:11:11.534738 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:11.534600 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7" event={"ID":"ec842943-f0f8-4fd0-b32d-96314b30077f","Type":"ContainerDied","Data":"09f3d06efa2cbd0be2ad5957d9ffa90d5011c6069e12b6bd5fc0b871ef9c19f7"} Nov 02 22:11:11.534738 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:11.534632 4935 scope.go:122] "RemoveContainer" containerID="d480510702914ea1cd9e9fb1b8cc72c1de5ba89a5f3c2d6fc08021faec521427" Nov 02 22:11:11.547613 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:11.547596 4935 scope.go:122] "RemoveContainer" containerID="d480510702914ea1cd9e9fb1b8cc72c1de5ba89a5f3c2d6fc08021faec521427" Nov 02 22:11:11.547885 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:11:11.547868 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d480510702914ea1cd9e9fb1b8cc72c1de5ba89a5f3c2d6fc08021faec521427\": container with ID starting with d480510702914ea1cd9e9fb1b8cc72c1de5ba89a5f3c2d6fc08021faec521427 not found: ID does not exist" containerID="d480510702914ea1cd9e9fb1b8cc72c1de5ba89a5f3c2d6fc08021faec521427" Nov 02 22:11:11.547949 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:11.547895 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d480510702914ea1cd9e9fb1b8cc72c1de5ba89a5f3c2d6fc08021faec521427"} err="failed to get container status \"d480510702914ea1cd9e9fb1b8cc72c1de5ba89a5f3c2d6fc08021faec521427\": rpc error: code = NotFound desc = could not find container \"d480510702914ea1cd9e9fb1b8cc72c1de5ba89a5f3c2d6fc08021faec521427\": container with ID starting with d480510702914ea1cd9e9fb1b8cc72c1de5ba89a5f3c2d6fc08021faec521427 not found: ID does not exist" Nov 02 22:11:11.690841 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:11.690802 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7"] Nov 02 22:11:11.735880 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:11.735850 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8702a9-5df5f49cf5-2nls7"] Nov 02 22:11:11.777711 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:11.777678 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec842943-f0f8-4fd0-b32d-96314b30077f" path="/var/lib/kubelet/pods/ec842943-f0f8-4fd0-b32d-96314b30077f/volumes" Nov 02 22:11:12.736418 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:12.736374 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:13.647668 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:13.647627 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:14.124905 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:14.124859 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:14.909632 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:14.909592 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:17.382690 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:17.382640 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:17.606657 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:17.606624 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:18.164836 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:18.164793 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:20.871905 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:20.871859 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:21.718676 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:21.718645 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:21.940221 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:21.940181 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:22.860180 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:22.860109 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj"] Nov 02 22:11:22.860371 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:22.860349 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj" podUID="67e33d3d-0533-44c0-842e-c5122d9a3dae" containerName="maas-api" containerID="cri-o://af5cf64ccfb51f90c6ec3b5123806b1f69c0294788b8e60c1ece22dc9f1e8eb8" gracePeriod=30 Nov 02 22:11:23.049438 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:23.049376 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:23.205461 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:23.205435 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj" Nov 02 22:11:23.352848 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:23.352294 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67e33d3d-0533-44c0-842e-c5122d9a3dae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67e33d3d-0533-44c0-842e-c5122d9a3dae-maas-api-tls\") pod \"67e33d3d-0533-44c0-842e-c5122d9a3dae\" (UID: \"67e33d3d-0533-44c0-842e-c5122d9a3dae\") " Nov 02 22:11:23.352848 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:23.352419 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67e33d3d-0533-44c0-842e-c5122d9a3dae-kube-api-access-8zbnf\" (UniqueName: \"kubernetes.io/projected/67e33d3d-0533-44c0-842e-c5122d9a3dae-kube-api-access-8zbnf\") pod \"67e33d3d-0533-44c0-842e-c5122d9a3dae\" (UID: \"67e33d3d-0533-44c0-842e-c5122d9a3dae\") " Nov 02 22:11:23.352848 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:23.352463 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67e33d3d-0533-44c0-842e-c5122d9a3dae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/67e33d3d-0533-44c0-842e-c5122d9a3dae-metrics-certs\") pod \"67e33d3d-0533-44c0-842e-c5122d9a3dae\" (UID: \"67e33d3d-0533-44c0-842e-c5122d9a3dae\") " Nov 02 22:11:23.352848 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:23.352562 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67e33d3d-0533-44c0-842e-c5122d9a3dae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e33d3d-0533-44c0-842e-c5122d9a3dae-openshift-service-ca\") pod \"67e33d3d-0533-44c0-842e-c5122d9a3dae\" (UID: \"67e33d3d-0533-44c0-842e-c5122d9a3dae\") " Nov 02 22:11:23.354662 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:23.354358 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e33d3d-0533-44c0-842e-c5122d9a3dae-openshift-service-ca" pod "67e33d3d-0533-44c0-842e-c5122d9a3dae" (UID: "67e33d3d-0533-44c0-842e-c5122d9a3dae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:11:23.357077 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:23.357044 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e33d3d-0533-44c0-842e-c5122d9a3dae-kube-api-access-8zbnf" pod "67e33d3d-0533-44c0-842e-c5122d9a3dae" (UID: "67e33d3d-0533-44c0-842e-c5122d9a3dae"). InnerVolumeSpecName "kube-api-access-8zbnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:11:23.357411 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:23.357376 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e33d3d-0533-44c0-842e-c5122d9a3dae-maas-api-tls" pod "67e33d3d-0533-44c0-842e-c5122d9a3dae" (UID: "67e33d3d-0533-44c0-842e-c5122d9a3dae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:11:23.371891 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:23.371852 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e33d3d-0533-44c0-842e-c5122d9a3dae-metrics-certs" pod "67e33d3d-0533-44c0-842e-c5122d9a3dae" (UID: "67e33d3d-0533-44c0-842e-c5122d9a3dae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:11:23.455632 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:23.455525 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67e33d3d-0533-44c0-842e-c5122d9a3dae-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:23.455632 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:23.455566 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zbnf\" (UniqueName: \"kubernetes.io/projected/67e33d3d-0533-44c0-842e-c5122d9a3dae-kube-api-access-8zbnf\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:23.455632 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:23.455589 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67e33d3d-0533-44c0-842e-c5122d9a3dae-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:23.455632 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:23.455603 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e33d3d-0533-44c0-842e-c5122d9a3dae-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:23.635907 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:23.635873 4935 generic.go:356] "Generic (PLEG): container finished" podID="67e33d3d-0533-44c0-842e-c5122d9a3dae" containerID="af5cf64ccfb51f90c6ec3b5123806b1f69c0294788b8e60c1ece22dc9f1e8eb8" exitCode=0 Nov 02 22:11:23.636100 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:23.635935 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj" Nov 02 22:11:23.636100 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:23.635965 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj" event={"ID":"67e33d3d-0533-44c0-842e-c5122d9a3dae","Type":"ContainerDied","Data":"af5cf64ccfb51f90c6ec3b5123806b1f69c0294788b8e60c1ece22dc9f1e8eb8"} Nov 02 22:11:23.636100 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:23.636049 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj" event={"ID":"67e33d3d-0533-44c0-842e-c5122d9a3dae","Type":"ContainerDied","Data":"db54041dacdbc7bf7b0267ed6b36d1c6894e6d31d9a5d7d7c61c1912ba4b4dc4"} Nov 02 22:11:23.636100 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:23.636081 4935 scope.go:122] "RemoveContainer" containerID="af5cf64ccfb51f90c6ec3b5123806b1f69c0294788b8e60c1ece22dc9f1e8eb8" Nov 02 22:11:23.648361 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:23.648341 4935 scope.go:122] "RemoveContainer" containerID="af5cf64ccfb51f90c6ec3b5123806b1f69c0294788b8e60c1ece22dc9f1e8eb8" Nov 02 22:11:23.648661 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:11:23.648638 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af5cf64ccfb51f90c6ec3b5123806b1f69c0294788b8e60c1ece22dc9f1e8eb8\": container with ID starting with af5cf64ccfb51f90c6ec3b5123806b1f69c0294788b8e60c1ece22dc9f1e8eb8 not found: ID does not exist" containerID="af5cf64ccfb51f90c6ec3b5123806b1f69c0294788b8e60c1ece22dc9f1e8eb8" Nov 02 22:11:23.648731 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:23.648669 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af5cf64ccfb51f90c6ec3b5123806b1f69c0294788b8e60c1ece22dc9f1e8eb8"} err="failed to get container status \"af5cf64ccfb51f90c6ec3b5123806b1f69c0294788b8e60c1ece22dc9f1e8eb8\": rpc error: code = NotFound desc = could not find container \"af5cf64ccfb51f90c6ec3b5123806b1f69c0294788b8e60c1ece22dc9f1e8eb8\": container with ID starting with af5cf64ccfb51f90c6ec3b5123806b1f69c0294788b8e60c1ece22dc9f1e8eb8 not found: ID does not exist" Nov 02 22:11:23.788702 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:23.788673 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj"] Nov 02 22:11:23.791424 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:23.791394 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6ba06f-78bcb6649b-hh5mj"] Nov 02 22:11:24.347336 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:24.347297 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:25.643377 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:25.643344 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:25.776705 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:25.776668 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67e33d3d-0533-44c0-842e-c5122d9a3dae" path="/var/lib/kubelet/pods/67e33d3d-0533-44c0-842e-c5122d9a3dae/volumes" Nov 02 22:11:26.230819 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:26.230784 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:26.449415 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:26.449370 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:26.727619 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:26.727577 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:27.570701 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:27.570666 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:27.791430 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:27.791394 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:28.104904 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:28.104850 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:28.471992 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:11:28.471901 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:11:28.850701 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:28.850661 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:29.070825 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.070788 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:29.554580 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.554542 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2"] Nov 02 22:11:29.555702 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.555674 4935 cpu_manager.go:414] "containerMap: removing container" podUID="67e33d3d-0533-44c0-842e-c5122d9a3dae" containerName="maas-api" Nov 02 22:11:29.556304 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.556279 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67e33d3d-0533-44c0-842e-c5122d9a3dae" containerName="maas-api" Nov 02 22:11:29.556518 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.556501 4935 cpu_manager.go:414] "containerMap: removing container" podUID="ec842943-f0f8-4fd0-b32d-96314b30077f" containerName="maas-api" Nov 02 22:11:29.556628 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.556616 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec842943-f0f8-4fd0-b32d-96314b30077f" containerName="maas-api" Nov 02 22:11:29.556879 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.556866 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="67e33d3d-0533-44c0-842e-c5122d9a3dae" containerName="maas-api" Nov 02 22:11:29.556994 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.556983 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec842943-f0f8-4fd0-b32d-96314b30077f" containerName="maas-api" Nov 02 22:11:29.565085 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.565057 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:29.644072 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.644027 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4585ee-openshift-default-dockercfg-zhzv5\"" Nov 02 22:11:29.644629 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.644598 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2"] Nov 02 22:11:29.655184 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.655153 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-workload-certs\") pod \"e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:29.655339 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.655215 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istio-data\") pod \"e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:29.655409 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.655331 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-workload-socket\") pod \"e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:29.655463 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.655415 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istio-envoy\") pod \"e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:29.655526 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.655461 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istio-podinfo\") pod \"e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:29.655587 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.655524 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-credential-socket\") pod \"e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:29.655587 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.655580 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istiod-ca-cert\") pod \"e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:29.655697 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.655622 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istio-token\") pod \"e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:29.655697 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.655647 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p29xr\" (UniqueName: \"kubernetes.io/projected/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-kube-api-access-p29xr\") pod \"e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:29.709946 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.709894 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:29.756691 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.756639 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istiod-ca-cert\") pod \"e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:29.756943 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.756713 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istio-token\") pod \"e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:29.756943 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.756742 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p29xr\" (UniqueName: \"kubernetes.io/projected/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-kube-api-access-p29xr\") pod \"e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:29.756943 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.756770 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-workload-certs\") pod \"e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:29.756943 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.756821 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istio-data\") pod \"e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:29.756943 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.756880 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-workload-socket\") pod \"e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:29.757373 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.756962 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istio-envoy\") pod \"e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:29.757373 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.757015 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istio-podinfo\") pod \"e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:29.757373 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.757088 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-credential-socket\") pod \"e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:29.757549 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.757497 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istio-data\") pod \"e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:29.757549 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.757509 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-credential-socket\") pod \"e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:29.757656 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.757560 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istiod-ca-cert\") pod \"e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:29.757772 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.757750 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-workload-socket\") pod \"e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:29.757837 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.757813 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-workload-certs\") pod \"e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:29.760342 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.760316 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istio-podinfo\") pod \"e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:29.760439 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.760321 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istio-envoy\") pod \"e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:29.813757 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.813678 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p29xr\" (UniqueName: \"kubernetes.io/projected/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-kube-api-access-p29xr\") pod \"e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:29.814711 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.814681 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istio-token\") pod \"e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:29.886982 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:29.886949 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:30.091548 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:30.091451 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:30.348514 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:30.348484 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2"] Nov 02 22:11:30.349857 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:11:30.349825 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc4fe754_4b3a_43d8_8a8f_12c49c48f936.slice/crio-6f2bfffcf8126588e70341fbe68f13bc1217cf2a29c598a8ef429945d741d37d WatchSource:0}: Error finding container 6f2bfffcf8126588e70341fbe68f13bc1217cf2a29c598a8ef429945d741d37d: Status 404 returned error can't find the container with id 6f2bfffcf8126588e70341fbe68f13bc1217cf2a29c598a8ef429945d741d37d Nov 02 22:11:30.352838 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:30.352804 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:11:30.352962 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:30.352876 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:11:30.352962 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:30.352903 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:11:30.380611 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:30.380584 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:30.598518 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:30.598416 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:30.722461 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:30.722407 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" event={"ID":"cc4fe754-4b3a-43d8-8a8f-12c49c48f936","Type":"ContainerStarted","Data":"30b8a87e6aec53b686771ab450689f797e97fec03614d000ce03ea4e8f1f9352"} Nov 02 22:11:30.722642 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:30.722487 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" event={"ID":"cc4fe754-4b3a-43d8-8a8f-12c49c48f936","Type":"ContainerStarted","Data":"6f2bfffcf8126588e70341fbe68f13bc1217cf2a29c598a8ef429945d741d37d"} Nov 02 22:11:30.836650 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:30.836585 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" podStartSLOduration=1.836564148 podStartE2EDuration="1.836564148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:30.833149701 +0000 UTC m=+3783.272724424" watchObservedRunningTime="2026-11-02 22:11:30.836564148 +0000 UTC m=+3783.276138871" Nov 02 22:11:30.887262 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:30.887193 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:31.391122 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:31.391082 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:31.893081 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:31.893051 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:31.926620 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:31.926583 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:32.736851 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:32.736816 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:32.738070 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:32.738049 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:11:33.046184 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:33.046108 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:33.742502 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:33.742458 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:33.979347 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:33.979310 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:35.104333 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:35.104301 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:36.706852 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:36.706812 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:36.919104 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:36.919060 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:38.602885 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:38.602836 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:38.821318 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:38.821280 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:39.949075 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:39.949041 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:40.173204 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:40.173171 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:41.372674 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:41.372636 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:43.505212 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:43.505180 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:44.121562 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:44.121524 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:49.252055 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:49.252001 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:49.474312 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:49.474272 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:49.695965 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:49.695934 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:55.021465 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:55.021429 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:55.248893 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:11:55.248853 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:58.469520 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:11:58.469482 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:12:00.322651 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:00.322610 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:05.286885 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:05.286853 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:10.424307 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:10.424068 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:15.444053 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:15.444016 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:19.161073 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:19.161034 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:19.395975 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:19.395929 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:20.111334 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:20.111296 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s"] Nov 02 22:12:20.116368 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:20.116343 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s" Nov 02 22:12:20.167634 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:20.167602 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s"] Nov 02 22:12:20.202776 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:20.202744 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7npr2\" (UniqueName: \"kubernetes.io/projected/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-kube-api-access-7npr2\") pod \"maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s\" (UID: \"8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s" Nov 02 22:12:20.202987 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:20.202897 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-openshift-service-ca\") pod \"maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s\" (UID: \"8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s" Nov 02 22:12:20.202987 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:20.202968 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-065be7-serving-cert\"" Nov 02 22:12:20.203093 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:20.202982 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-maas-api-tls\") pod \"maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s\" (UID: \"8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s" Nov 02 22:12:20.203093 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:20.203020 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-metrics-certs\") pod \"maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s\" (UID: \"8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s" Nov 02 22:12:20.304805 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:20.304762 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7npr2\" (UniqueName: \"kubernetes.io/projected/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-kube-api-access-7npr2\") pod \"maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s\" (UID: \"8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s" Nov 02 22:12:20.304973 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:20.304903 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-openshift-service-ca\") pod \"maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s\" (UID: \"8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s" Nov 02 22:12:20.304973 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:20.304955 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-maas-api-tls\") pod \"maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s\" (UID: \"8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s" Nov 02 22:12:20.305075 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:12:20.305049 4935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-serving-cert: secret "maas-api-e2e-inf-a-065be7-serving-cert" not found Nov 02 22:12:20.305143 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:20.305081 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-metrics-certs\") pod \"maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s\" (UID: \"8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s" Nov 02 22:12:20.305143 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:12:20.305121 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-maas-api-tls podName:8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39 nodeName:}" failed. No retries permitted until 2026-11-02 22:12:20.805101684 +0000 UTC m=+3833.244676384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-maas-api-tls") pod "maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s" (UID: "8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39") : secret "maas-api-e2e-inf-a-065be7-serving-cert" not found Nov 02 22:12:20.305597 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:20.305577 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-openshift-service-ca\") pod \"maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s\" (UID: \"8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s" Nov 02 22:12:20.307736 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:20.307710 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-metrics-certs\") pod \"maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s\" (UID: \"8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s" Nov 02 22:12:20.357982 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:20.357944 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7npr2\" (UniqueName: \"kubernetes.io/projected/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-kube-api-access-7npr2\") pod \"maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s\" (UID: \"8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s" Nov 02 22:12:20.388387 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:20.388304 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:20.607558 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:20.607508 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:20.810870 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:20.810824 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-maas-api-tls\") pod \"maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s\" (UID: \"8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s" Nov 02 22:12:20.813392 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:20.813366 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-maas-api-tls\") pod \"maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s\" (UID: \"8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s" Nov 02 22:12:21.031470 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:21.031438 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s" Nov 02 22:12:21.402007 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:21.401974 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s"] Nov 02 22:12:21.407562 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:12:21.407529 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c7f7bb3_ec3c_444e_80f5_60d71cfcbc39.slice/crio-3ff5f9147e45720c1277f7fa9805b495d83a61fdcf92f67baf0797b7ba9713b6 WatchSource:0}: Error finding container 3ff5f9147e45720c1277f7fa9805b495d83a61fdcf92f67baf0797b7ba9713b6: Status 404 returned error can't find the container with id 3ff5f9147e45720c1277f7fa9805b495d83a61fdcf92f67baf0797b7ba9713b6 Nov 02 22:12:21.408634 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:21.408611 4935 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:12:22.075264 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:22.075225 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s" event={"ID":"8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39","Type":"ContainerStarted","Data":"3ff5f9147e45720c1277f7fa9805b495d83a61fdcf92f67baf0797b7ba9713b6"} Nov 02 22:12:23.082580 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:23.082541 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s" event={"ID":"8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39","Type":"ContainerStarted","Data":"1c2b15fcee9a6e80af0de086e428d25a53da39cd5d09481535c234ea591e4021"} Nov 02 22:12:23.082991 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:23.082641 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s" Nov 02 22:12:23.190441 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:23.190280 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s" podStartSLOduration=2.221019611 podStartE2EDuration="3.190261534s" totalImagesPullingTime="969.241923ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:23.18896219 +0000 UTC m=+3835.628536913" watchObservedRunningTime="2026-11-02 22:12:23.190261534 +0000 UTC m=+3835.629836252" Nov 02 22:12:25.557601 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:25.557539 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:28.470226 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:12:28.470188 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:12:29.093937 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:29.093900 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s" Nov 02 22:12:30.553790 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:30.553754 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:35.522063 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:35.522022 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:38.931978 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:38.931943 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:40.565399 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:40.565359 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:45.555442 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:45.555392 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:50.932262 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:50.932229 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:57.023925 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:57.023878 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:57.244660 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:57.244624 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:57.399966 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:57.399880 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq"] Nov 02 22:12:57.400123 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:57.400096 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" podUID="8bf1a756-7259-404a-aa43-ca017939d6ec" containerName="istio-proxy" containerID="cri-o://a91e1b8e72446adc8ab4844f64d594582d0cdad4d3324d5a8c5a54b3ed2ba821" gracePeriod=30 Nov 02 22:12:57.488478 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:57.488446 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:58.391951 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:58.391917 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:58.468617 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:12:58.468564 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:12:58.974957 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:58.974889 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:59.249678 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:59.249586 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:59.470398 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:12:59.470355 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:02.134305 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.134271 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86"] Nov 02 22:13:02.138771 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.138747 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86" Nov 02 22:13:02.187503 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.187465 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-4585ee-serving-cert\"" Nov 02 22:13:02.238933 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.238893 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86"] Nov 02 22:13:02.280623 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.280582 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f4cfd18-8663-4bb1-8452-595d606cca1c-maas-api-tls\") pod \"maas-api-e2e-inf-a-4585ee-579dffd776-6lp86\" (UID: \"9f4cfd18-8663-4bb1-8452-595d606cca1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86" Nov 02 22:13:02.280834 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.280781 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f4cfd18-8663-4bb1-8452-595d606cca1c-metrics-certs\") pod \"maas-api-e2e-inf-a-4585ee-579dffd776-6lp86\" (UID: \"9f4cfd18-8663-4bb1-8452-595d606cca1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86" Nov 02 22:13:02.280926 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.280890 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7fr2\" (UniqueName: \"kubernetes.io/projected/9f4cfd18-8663-4bb1-8452-595d606cca1c-kube-api-access-p7fr2\") pod \"maas-api-e2e-inf-a-4585ee-579dffd776-6lp86\" (UID: \"9f4cfd18-8663-4bb1-8452-595d606cca1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86" Nov 02 22:13:02.281067 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.280993 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f4cfd18-8663-4bb1-8452-595d606cca1c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4585ee-579dffd776-6lp86\" (UID: \"9f4cfd18-8663-4bb1-8452-595d606cca1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86" Nov 02 22:13:02.381996 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.381959 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f4cfd18-8663-4bb1-8452-595d606cca1c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4585ee-579dffd776-6lp86\" (UID: \"9f4cfd18-8663-4bb1-8452-595d606cca1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86" Nov 02 22:13:02.382199 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.382053 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f4cfd18-8663-4bb1-8452-595d606cca1c-maas-api-tls\") pod \"maas-api-e2e-inf-a-4585ee-579dffd776-6lp86\" (UID: \"9f4cfd18-8663-4bb1-8452-595d606cca1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86" Nov 02 22:13:02.382199 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.382121 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f4cfd18-8663-4bb1-8452-595d606cca1c-metrics-certs\") pod \"maas-api-e2e-inf-a-4585ee-579dffd776-6lp86\" (UID: \"9f4cfd18-8663-4bb1-8452-595d606cca1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86" Nov 02 22:13:02.382298 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.382201 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7fr2\" (UniqueName: \"kubernetes.io/projected/9f4cfd18-8663-4bb1-8452-595d606cca1c-kube-api-access-p7fr2\") pod \"maas-api-e2e-inf-a-4585ee-579dffd776-6lp86\" (UID: \"9f4cfd18-8663-4bb1-8452-595d606cca1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86" Nov 02 22:13:02.382298 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:13:02.382226 4935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-serving-cert: secret "maas-api-e2e-inf-a-4585ee-serving-cert" not found Nov 02 22:13:02.382382 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:13:02.382302 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f4cfd18-8663-4bb1-8452-595d606cca1c-maas-api-tls podName:9f4cfd18-8663-4bb1-8452-595d606cca1c nodeName:}" failed. No retries permitted until 2026-11-02 22:13:02.882281395 +0000 UTC m=+3875.321856117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9f4cfd18-8663-4bb1-8452-595d606cca1c-maas-api-tls") pod "maas-api-e2e-inf-a-4585ee-579dffd776-6lp86" (UID: "9f4cfd18-8663-4bb1-8452-595d606cca1c") : secret "maas-api-e2e-inf-a-4585ee-serving-cert" not found Nov 02 22:13:02.382745 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.382723 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f4cfd18-8663-4bb1-8452-595d606cca1c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4585ee-579dffd776-6lp86\" (UID: \"9f4cfd18-8663-4bb1-8452-595d606cca1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86" Nov 02 22:13:02.385192 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.384862 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f4cfd18-8663-4bb1-8452-595d606cca1c-metrics-certs\") pod \"maas-api-e2e-inf-a-4585ee-579dffd776-6lp86\" (UID: \"9f4cfd18-8663-4bb1-8452-595d606cca1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86" Nov 02 22:13:02.424521 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.424482 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:02.438094 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.438061 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7fr2\" (UniqueName: \"kubernetes.io/projected/9f4cfd18-8663-4bb1-8452-595d606cca1c-kube-api-access-p7fr2\") pod \"maas-api-e2e-inf-a-4585ee-579dffd776-6lp86\" (UID: \"9f4cfd18-8663-4bb1-8452-595d606cca1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86" Nov 02 22:13:02.637299 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.637210 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:02.690075 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.690049 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:13:02.785675 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.785640 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bf1a756-7259-404a-aa43-ca017939d6ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf1a756-7259-404a-aa43-ca017939d6ec-istiod-ca-cert\") pod \"8bf1a756-7259-404a-aa43-ca017939d6ec\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " Nov 02 22:13:02.785884 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.785708 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-workload-certs\") pod \"8bf1a756-7259-404a-aa43-ca017939d6ec\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " Nov 02 22:13:02.785884 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.785734 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-credential-socket\") pod \"8bf1a756-7259-404a-aa43-ca017939d6ec\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " Nov 02 22:13:02.785884 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.785797 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8bf1a756-7259-404a-aa43-ca017939d6ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf1a756-7259-404a-aa43-ca017939d6ec-istio-podinfo\") pod \"8bf1a756-7259-404a-aa43-ca017939d6ec\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " Nov 02 22:13:02.785884 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.785836 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-istio-envoy\") pod \"8bf1a756-7259-404a-aa43-ca017939d6ec\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " Nov 02 22:13:02.785884 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.785873 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf1a756-7259-404a-aa43-ca017939d6ec-kube-api-access-d2fhq\" (UniqueName: \"kubernetes.io/projected/8bf1a756-7259-404a-aa43-ca017939d6ec-kube-api-access-d2fhq\") pod \"8bf1a756-7259-404a-aa43-ca017939d6ec\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " Nov 02 22:13:02.786127 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.785914 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf1a756-7259-404a-aa43-ca017939d6ec-istio-token\" (UniqueName: \"kubernetes.io/projected/8bf1a756-7259-404a-aa43-ca017939d6ec-istio-token\") pod \"8bf1a756-7259-404a-aa43-ca017939d6ec\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " Nov 02 22:13:02.786127 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.785988 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-workload-socket\") pod \"8bf1a756-7259-404a-aa43-ca017939d6ec\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " Nov 02 22:13:02.786127 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.786004 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-workload-certs" pod "8bf1a756-7259-404a-aa43-ca017939d6ec" (UID: "8bf1a756-7259-404a-aa43-ca017939d6ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:13:02.786127 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.786016 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-istio-data\") pod \"8bf1a756-7259-404a-aa43-ca017939d6ec\" (UID: \"8bf1a756-7259-404a-aa43-ca017939d6ec\") " Nov 02 22:13:02.786127 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.786061 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf1a756-7259-404a-aa43-ca017939d6ec-istiod-ca-cert" pod "8bf1a756-7259-404a-aa43-ca017939d6ec" (UID: "8bf1a756-7259-404a-aa43-ca017939d6ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:02.786437 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.786208 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-credential-socket" pod "8bf1a756-7259-404a-aa43-ca017939d6ec" (UID: "8bf1a756-7259-404a-aa43-ca017939d6ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:13:02.786437 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.786419 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-workload-socket" pod "8bf1a756-7259-404a-aa43-ca017939d6ec" (UID: "8bf1a756-7259-404a-aa43-ca017939d6ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:13:02.786558 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.786488 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-istio-data" pod "8bf1a756-7259-404a-aa43-ca017939d6ec" (UID: "8bf1a756-7259-404a-aa43-ca017939d6ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:13:02.786680 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.786656 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf1a756-7259-404a-aa43-ca017939d6ec-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:02.786735 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.786688 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:02.786735 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.786704 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:02.786735 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.786717 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:02.786735 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.786733 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:02.788294 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.788267 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf1a756-7259-404a-aa43-ca017939d6ec-istio-token" pod "8bf1a756-7259-404a-aa43-ca017939d6ec" (UID: "8bf1a756-7259-404a-aa43-ca017939d6ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:02.788393 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.788359 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf1a756-7259-404a-aa43-ca017939d6ec-kube-api-access-d2fhq" pod "8bf1a756-7259-404a-aa43-ca017939d6ec" (UID: "8bf1a756-7259-404a-aa43-ca017939d6ec"). InnerVolumeSpecName "kube-api-access-d2fhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:02.788639 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.788622 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-istio-envoy" pod "8bf1a756-7259-404a-aa43-ca017939d6ec" (UID: "8bf1a756-7259-404a-aa43-ca017939d6ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:13:02.788683 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.788627 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bf1a756-7259-404a-aa43-ca017939d6ec-istio-podinfo" pod "8bf1a756-7259-404a-aa43-ca017939d6ec" (UID: "8bf1a756-7259-404a-aa43-ca017939d6ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:13:02.855580 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.855537 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:02.888225 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.888141 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f4cfd18-8663-4bb1-8452-595d606cca1c-maas-api-tls\") pod \"maas-api-e2e-inf-a-4585ee-579dffd776-6lp86\" (UID: \"9f4cfd18-8663-4bb1-8452-595d606cca1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86" Nov 02 22:13:02.888684 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.888654 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf1a756-7259-404a-aa43-ca017939d6ec-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:02.888684 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.888685 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf1a756-7259-404a-aa43-ca017939d6ec-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:02.888870 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.888699 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2fhq\" (UniqueName: \"kubernetes.io/projected/8bf1a756-7259-404a-aa43-ca017939d6ec-kube-api-access-d2fhq\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:02.888870 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.888715 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf1a756-7259-404a-aa43-ca017939d6ec-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:02.890774 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:02.890751 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f4cfd18-8663-4bb1-8452-595d606cca1c-maas-api-tls\") pod \"maas-api-e2e-inf-a-4585ee-579dffd776-6lp86\" (UID: \"9f4cfd18-8663-4bb1-8452-595d606cca1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86" Nov 02 22:13:03.052304 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:03.052264 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86" Nov 02 22:13:03.092316 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:03.092279 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:03.355366 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:03.355333 4935 generic.go:356] "Generic (PLEG): container finished" podID="8bf1a756-7259-404a-aa43-ca017939d6ec" containerID="a91e1b8e72446adc8ab4844f64d594582d0cdad4d3324d5a8c5a54b3ed2ba821" exitCode=0 Nov 02 22:13:03.355840 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:03.355392 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" Nov 02 22:13:03.355840 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:03.355385 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" event={"ID":"8bf1a756-7259-404a-aa43-ca017939d6ec","Type":"ContainerDied","Data":"a91e1b8e72446adc8ab4844f64d594582d0cdad4d3324d5a8c5a54b3ed2ba821"} Nov 02 22:13:03.355840 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:03.355516 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq" event={"ID":"8bf1a756-7259-404a-aa43-ca017939d6ec","Type":"ContainerDied","Data":"255e3fe4b9ff6e2a1dbc46e4461cf9dc398ef1a6bf8727338d3b61f127c3efc2"} Nov 02 22:13:03.355840 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:03.355545 4935 scope.go:122] "RemoveContainer" containerID="a91e1b8e72446adc8ab4844f64d594582d0cdad4d3324d5a8c5a54b3ed2ba821" Nov 02 22:13:03.361411 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:03.361377 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:03.369789 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:03.369767 4935 scope.go:122] "RemoveContainer" containerID="6fc12406f086170f8273384cbc391acb7eefc1110c6355db4b213851afb3c4c6" Nov 02 22:13:03.380877 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:03.380858 4935 scope.go:122] "RemoveContainer" containerID="a91e1b8e72446adc8ab4844f64d594582d0cdad4d3324d5a8c5a54b3ed2ba821" Nov 02 22:13:03.381205 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:13:03.381178 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a91e1b8e72446adc8ab4844f64d594582d0cdad4d3324d5a8c5a54b3ed2ba821\": container with ID starting with a91e1b8e72446adc8ab4844f64d594582d0cdad4d3324d5a8c5a54b3ed2ba821 not found: ID does not exist" containerID="a91e1b8e72446adc8ab4844f64d594582d0cdad4d3324d5a8c5a54b3ed2ba821" Nov 02 22:13:03.381275 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:03.381218 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a91e1b8e72446adc8ab4844f64d594582d0cdad4d3324d5a8c5a54b3ed2ba821"} err="failed to get container status \"a91e1b8e72446adc8ab4844f64d594582d0cdad4d3324d5a8c5a54b3ed2ba821\": rpc error: code = NotFound desc = could not find container \"a91e1b8e72446adc8ab4844f64d594582d0cdad4d3324d5a8c5a54b3ed2ba821\": container with ID starting with a91e1b8e72446adc8ab4844f64d594582d0cdad4d3324d5a8c5a54b3ed2ba821 not found: ID does not exist" Nov 02 22:13:03.381275 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:03.381235 4935 scope.go:122] "RemoveContainer" containerID="6fc12406f086170f8273384cbc391acb7eefc1110c6355db4b213851afb3c4c6" Nov 02 22:13:03.381506 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:13:03.381487 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fc12406f086170f8273384cbc391acb7eefc1110c6355db4b213851afb3c4c6\": container with ID starting with 6fc12406f086170f8273384cbc391acb7eefc1110c6355db4b213851afb3c4c6 not found: ID does not exist" containerID="6fc12406f086170f8273384cbc391acb7eefc1110c6355db4b213851afb3c4c6" Nov 02 22:13:03.381547 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:03.381513 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fc12406f086170f8273384cbc391acb7eefc1110c6355db4b213851afb3c4c6"} err="failed to get container status \"6fc12406f086170f8273384cbc391acb7eefc1110c6355db4b213851afb3c4c6\": rpc error: code = NotFound desc = could not find container \"6fc12406f086170f8273384cbc391acb7eefc1110c6355db4b213851afb3c4c6\": container with ID starting with 6fc12406f086170f8273384cbc391acb7eefc1110c6355db4b213851afb3c4c6 not found: ID does not exist" Nov 02 22:13:03.429124 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:03.429095 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86"] Nov 02 22:13:03.433280 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:13:03.433246 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4cfd18_8663_4bb1_8452_595d606cca1c.slice/crio-9f35cc98029bc4f9c86d10772ee3a11190036162f5c5f8bed0f9271d692c95bc WatchSource:0}: Error finding container 9f35cc98029bc4f9c86d10772ee3a11190036162f5c5f8bed0f9271d692c95bc: Status 404 returned error can't find the container with id 9f35cc98029bc4f9c86d10772ee3a11190036162f5c5f8bed0f9271d692c95bc Nov 02 22:13:03.510708 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:03.510674 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq"] Nov 02 22:13:03.524251 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:03.523527 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-6ba06f-openshift-default-84d4676bdf-cqkkq"] Nov 02 22:13:03.776486 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:03.776442 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bf1a756-7259-404a-aa43-ca017939d6ec" path="/var/lib/kubelet/pods/8bf1a756-7259-404a-aa43-ca017939d6ec/volumes" Nov 02 22:13:04.365219 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:04.365178 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86" event={"ID":"9f4cfd18-8663-4bb1-8452-595d606cca1c","Type":"ContainerStarted","Data":"9f35cc98029bc4f9c86d10772ee3a11190036162f5c5f8bed0f9271d692c95bc"} Nov 02 22:13:05.375870 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:05.375816 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86" event={"ID":"9f4cfd18-8663-4bb1-8452-595d606cca1c","Type":"ContainerStarted","Data":"54dcb62e0fcb11295a03a65d9c48b4eab216a7916a6f65c3c69575362d5424e2"} Nov 02 22:13:05.376297 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:05.375915 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86" Nov 02 22:13:05.482993 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:05.482942 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86" podStartSLOduration=2.541087043 podStartE2EDuration="3.482926575s" totalImagesPullingTime="941.839532ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:05.481777189 +0000 UTC m=+3877.921351911" watchObservedRunningTime="2026-11-02 22:13:05.482926575 +0000 UTC m=+3877.922501297" Nov 02 22:13:06.241260 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:06.241223 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:08.159831 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:08.159792 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:09.442435 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:09.442395 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:09.670708 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:09.670630 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:09.900191 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:09.900146 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:11.325541 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:11.325502 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:11.389588 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:11.389557 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86" Nov 02 22:13:11.837789 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:11.837734 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:13.468748 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:13.468709 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:13.687266 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:13.687234 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:14.478886 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:14.478844 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:14.793860 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:14.793822 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:15.083445 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:15.083352 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:18.238064 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:18.238026 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:18.466258 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:18.466220 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:20.605087 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:20.605050 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:21.379367 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:21.379329 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:21.595632 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:21.595589 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:25.603897 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:25.603864 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:25.869593 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:25.869502 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:26.119229 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:26.119167 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:28.469675 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:13:28.469635 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:13:28.502694 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:28.502662 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 22:13:28.511631 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:28.511610 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 22:13:28.525679 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:28.525652 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 22:13:28.543747 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:28.543723 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 22:13:31.604769 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:31.604727 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:36.646868 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:36.646821 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:39.252040 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:39.251946 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:41.802110 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:41.802077 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:46.977660 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:46.977570 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:48.779341 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:48.779282 4935 patch_prober.go:29] interesting pod/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:13:48.779884 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:48.779368 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" podUID="cc4fe754-4b3a-43d8-8a8f-12c49c48f936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:50.721181 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:50.721112 4935 generic.go:356] "Generic (PLEG): container finished" podID="cc4fe754-4b3a-43d8-8a8f-12c49c48f936" containerID="30b8a87e6aec53b686771ab450689f797e97fec03614d000ce03ea4e8f1f9352" exitCode=0 Nov 02 22:13:50.721181 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:50.721174 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" event={"ID":"cc4fe754-4b3a-43d8-8a8f-12c49c48f936","Type":"ContainerDied","Data":"30b8a87e6aec53b686771ab450689f797e97fec03614d000ce03ea4e8f1f9352"} Nov 02 22:13:50.721701 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:50.721554 4935 scope.go:122] "RemoveContainer" containerID="30b8a87e6aec53b686771ab450689f797e97fec03614d000ce03ea4e8f1f9352" Nov 02 22:13:50.723050 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:50.723020 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:13:50.723178 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:50.723089 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:13:50.723178 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:50.723121 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:13:51.754157 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:51.753283 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" event={"ID":"cc4fe754-4b3a-43d8-8a8f-12c49c48f936","Type":"ContainerStarted","Data":"f2aa75c9eb022c0b13d0d3c89213e83204ad8d84a8cc57da3cddd017ad8c3e42"} Nov 02 22:13:51.887554 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:51.887516 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:13:51.887738 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:51.887712 4935 patch_prober.go:29] interesting pod/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 02 22:13:51.887794 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:51.887769 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" podUID="cc4fe754-4b3a-43d8-8a8f-12c49c48f936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 02 22:13:51.964240 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:51.964193 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:52.884239 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:52.884198 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:52.887282 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:52.887253 4935 patch_prober.go:29] interesting pod/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 02 22:13:52.887426 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:52.887301 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" podUID="cc4fe754-4b3a-43d8-8a8f-12c49c48f936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 02 22:13:53.887465 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:53.887432 4935 patch_prober.go:29] interesting pod/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 02 22:13:53.887881 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:53.887480 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" podUID="cc4fe754-4b3a-43d8-8a8f-12c49c48f936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 02 22:13:54.888293 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:54.888256 4935 patch_prober.go:29] interesting pod/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 02 22:13:54.888737 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:54.888315 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" podUID="cc4fe754-4b3a-43d8-8a8f-12c49c48f936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 02 22:13:55.887684 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:55.887652 4935 patch_prober.go:29] interesting pod/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 02 22:13:55.887875 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:55.887704 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" podUID="cc4fe754-4b3a-43d8-8a8f-12c49c48f936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 02 22:13:56.945485 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:56.945449 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:13:56.945990 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:56.945929 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:13:56.946998 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:56.946977 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:13:57.449784 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:57.449743 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:58.444364 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:58.444323 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:58.471398 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:13:58.471359 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:13:58.665509 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:58.665472 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:58.882872 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:13:58.882829 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:02.747655 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:02.747609 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:04.458468 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:04.458429 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:07.722906 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:07.722870 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:12.680827 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:12.680789 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:17.774509 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:17.774460 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:22.741703 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:22.741671 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:27.295214 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:27.295179 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh"] Nov 02 22:14:27.295710 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:27.295695 4935 cpu_manager.go:414] "containerMap: removing container" podUID="8bf1a756-7259-404a-aa43-ca017939d6ec" containerName="istio-proxy" Nov 02 22:14:27.295760 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:27.295711 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf1a756-7259-404a-aa43-ca017939d6ec" containerName="istio-proxy" Nov 02 22:14:27.295826 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:27.295817 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf1a756-7259-404a-aa43-ca017939d6ec" containerName="istio-proxy" Nov 02 22:14:27.299632 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:27.299610 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh" Nov 02 22:14:27.388753 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:27.388716 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwlsk\" (UniqueName: \"kubernetes.io/projected/94192fa7-1b0f-4384-9207-b0caf7e4eac7-kube-api-access-zwlsk\") pod \"maas-api-e2e-inf-a-72f279-5784755887-vlpwh\" (UID: \"94192fa7-1b0f-4384-9207-b0caf7e4eac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh" Nov 02 22:14:27.388925 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:27.388789 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94192fa7-1b0f-4384-9207-b0caf7e4eac7-maas-api-tls\") pod \"maas-api-e2e-inf-a-72f279-5784755887-vlpwh\" (UID: \"94192fa7-1b0f-4384-9207-b0caf7e4eac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh" Nov 02 22:14:27.388925 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:27.388909 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94192fa7-1b0f-4384-9207-b0caf7e4eac7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-72f279-5784755887-vlpwh\" (UID: \"94192fa7-1b0f-4384-9207-b0caf7e4eac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh" Nov 02 22:14:27.389015 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:27.389004 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94192fa7-1b0f-4384-9207-b0caf7e4eac7-metrics-certs\") pod \"maas-api-e2e-inf-a-72f279-5784755887-vlpwh\" (UID: \"94192fa7-1b0f-4384-9207-b0caf7e4eac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh" Nov 02 22:14:27.395888 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:27.395864 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-72f279-serving-cert\"" Nov 02 22:14:27.399204 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:27.399176 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh"] Nov 02 22:14:27.489676 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:27.489644 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94192fa7-1b0f-4384-9207-b0caf7e4eac7-metrics-certs\") pod \"maas-api-e2e-inf-a-72f279-5784755887-vlpwh\" (UID: \"94192fa7-1b0f-4384-9207-b0caf7e4eac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh" Nov 02 22:14:27.489870 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:27.489703 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwlsk\" (UniqueName: \"kubernetes.io/projected/94192fa7-1b0f-4384-9207-b0caf7e4eac7-kube-api-access-zwlsk\") pod \"maas-api-e2e-inf-a-72f279-5784755887-vlpwh\" (UID: \"94192fa7-1b0f-4384-9207-b0caf7e4eac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh" Nov 02 22:14:27.489870 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:27.489747 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94192fa7-1b0f-4384-9207-b0caf7e4eac7-maas-api-tls\") pod \"maas-api-e2e-inf-a-72f279-5784755887-vlpwh\" (UID: \"94192fa7-1b0f-4384-9207-b0caf7e4eac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh" Nov 02 22:14:27.489870 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:14:27.489864 4935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-serving-cert: secret "maas-api-e2e-inf-a-72f279-serving-cert" not found Nov 02 22:14:27.490020 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:14:27.489920 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94192fa7-1b0f-4384-9207-b0caf7e4eac7-maas-api-tls podName:94192fa7-1b0f-4384-9207-b0caf7e4eac7 nodeName:}" failed. No retries permitted until 2026-11-02 22:14:27.989904148 +0000 UTC m=+3960.429478849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/94192fa7-1b0f-4384-9207-b0caf7e4eac7-maas-api-tls") pod "maas-api-e2e-inf-a-72f279-5784755887-vlpwh" (UID: "94192fa7-1b0f-4384-9207-b0caf7e4eac7") : secret "maas-api-e2e-inf-a-72f279-serving-cert" not found Nov 02 22:14:27.490020 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:27.489945 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94192fa7-1b0f-4384-9207-b0caf7e4eac7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-72f279-5784755887-vlpwh\" (UID: \"94192fa7-1b0f-4384-9207-b0caf7e4eac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh" Nov 02 22:14:27.490566 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:27.490540 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94192fa7-1b0f-4384-9207-b0caf7e4eac7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-72f279-5784755887-vlpwh\" (UID: \"94192fa7-1b0f-4384-9207-b0caf7e4eac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh" Nov 02 22:14:27.492142 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:27.492105 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94192fa7-1b0f-4384-9207-b0caf7e4eac7-metrics-certs\") pod \"maas-api-e2e-inf-a-72f279-5784755887-vlpwh\" (UID: \"94192fa7-1b0f-4384-9207-b0caf7e4eac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh" Nov 02 22:14:27.543077 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:27.543028 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwlsk\" (UniqueName: \"kubernetes.io/projected/94192fa7-1b0f-4384-9207-b0caf7e4eac7-kube-api-access-zwlsk\") pod \"maas-api-e2e-inf-a-72f279-5784755887-vlpwh\" (UID: \"94192fa7-1b0f-4384-9207-b0caf7e4eac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh" Nov 02 22:14:27.592960 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:27.592860 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:27.833153 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:27.833087 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:27.996397 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:27.996319 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94192fa7-1b0f-4384-9207-b0caf7e4eac7-maas-api-tls\") pod \"maas-api-e2e-inf-a-72f279-5784755887-vlpwh\" (UID: \"94192fa7-1b0f-4384-9207-b0caf7e4eac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh" Nov 02 22:14:27.998882 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:27.998849 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94192fa7-1b0f-4384-9207-b0caf7e4eac7-maas-api-tls\") pod \"maas-api-e2e-inf-a-72f279-5784755887-vlpwh\" (UID: \"94192fa7-1b0f-4384-9207-b0caf7e4eac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh" Nov 02 22:14:28.045475 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:28.045446 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:28.213224 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:28.213194 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh" Nov 02 22:14:28.470096 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:14:28.470058 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:14:28.584161 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:14:28.584110 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94192fa7_1b0f_4384_9207_b0caf7e4eac7.slice/crio-38e54e5e17d93146d08f8d9ced32ffdfec84d776d0a80d48645f5485f8857315 WatchSource:0}: Error finding container 38e54e5e17d93146d08f8d9ced32ffdfec84d776d0a80d48645f5485f8857315: Status 404 returned error can't find the container with id 38e54e5e17d93146d08f8d9ced32ffdfec84d776d0a80d48645f5485f8857315 Nov 02 22:14:28.584687 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:28.584655 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh"] Nov 02 22:14:28.999698 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:28.999616 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh" event={"ID":"94192fa7-1b0f-4384-9207-b0caf7e4eac7","Type":"ContainerStarted","Data":"38e54e5e17d93146d08f8d9ced32ffdfec84d776d0a80d48645f5485f8857315"} Nov 02 22:14:30.007997 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:30.007903 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh" event={"ID":"94192fa7-1b0f-4384-9207-b0caf7e4eac7","Type":"ContainerStarted","Data":"f8c60b3b1711ad7fbaf20ba8c13a6a8e48fe074652c6c61d5324d87cb0177533"} Nov 02 22:14:30.008384 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:30.007998 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh" Nov 02 22:14:30.115236 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:30.115181 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh" podStartSLOduration=2.091619794 podStartE2EDuration="3.115168882s" totalImagesPullingTime="1.023549088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:30.114242352 +0000 UTC m=+3962.553817085" watchObservedRunningTime="2026-11-02 22:14:30.115168882 +0000 UTC m=+3962.554743604" Nov 02 22:14:32.771962 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:32.771928 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:36.021426 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:36.021395 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh" Nov 02 22:14:37.202701 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:37.202650 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:37.444099 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:37.444060 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:38.984454 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:38.984418 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:39.204683 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:39.204648 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:39.418725 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:39.418691 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94"] Nov 02 22:14:39.418967 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:39.418940 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" podUID="d9e7ef62-e7ba-474a-9f7d-650b4c98d0de" containerName="istio-proxy" containerID="cri-o://990cc840cc13c6ae300573492bb9de83a243de0b3d2f8e0f59fc113c5efc5203" gracePeriod=30 Nov 02 22:14:39.513377 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:39.513344 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:39.751433 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:39.751347 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:40.305089 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:40.305044 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:41.118013 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:41.117974 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:41.338914 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:41.338876 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:43.199491 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:43.199457 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:44.569236 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.569183 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:44.722028 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.722001 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:14:44.722438 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.722415 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-workload-socket\") pod \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " Nov 02 22:14:44.722502 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.722484 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-credential-socket\") pod \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " Nov 02 22:14:44.722543 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.722510 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istio-envoy\") pod \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " Nov 02 22:14:44.722584 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.722564 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-kube-api-access-k9xrs\" (UniqueName: \"kubernetes.io/projected/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-kube-api-access-k9xrs\") pod \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " Nov 02 22:14:44.722635 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.722594 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istio-podinfo\") pod \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " Nov 02 22:14:44.722694 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.722638 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-workload-certs\") pod \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " Nov 02 22:14:44.722756 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.722709 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-workload-socket" pod "d9e7ef62-e7ba-474a-9f7d-650b4c98d0de" (UID: "d9e7ef62-e7ba-474a-9f7d-650b4c98d0de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:44.722756 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.722737 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-credential-socket" pod "d9e7ef62-e7ba-474a-9f7d-650b4c98d0de" (UID: "d9e7ef62-e7ba-474a-9f7d-650b4c98d0de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:44.722860 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.722758 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istio-data\") pod \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " Nov 02 22:14:44.722925 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.722883 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-workload-certs" pod "d9e7ef62-e7ba-474a-9f7d-650b4c98d0de" (UID: "d9e7ef62-e7ba-474a-9f7d-650b4c98d0de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:44.722984 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.722885 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istio-token\" (UniqueName: \"kubernetes.io/projected/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istio-token\") pod \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " Nov 02 22:14:44.722984 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.722950 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istio-data" pod "d9e7ef62-e7ba-474a-9f7d-650b4c98d0de" (UID: "d9e7ef62-e7ba-474a-9f7d-650b4c98d0de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:44.723094 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.722986 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istiod-ca-cert\") pod \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\" (UID: \"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de\") " Nov 02 22:14:44.723400 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.723374 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istiod-ca-cert" pod "d9e7ef62-e7ba-474a-9f7d-650b4c98d0de" (UID: "d9e7ef62-e7ba-474a-9f7d-650b4c98d0de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:44.723565 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.723490 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:44.723691 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.723568 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:44.723691 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.723581 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:44.723691 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.723593 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:44.725083 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.725032 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istio-token" pod "d9e7ef62-e7ba-474a-9f7d-650b4c98d0de" (UID: "d9e7ef62-e7ba-474a-9f7d-650b4c98d0de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:44.725336 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.725309 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istio-envoy" pod "d9e7ef62-e7ba-474a-9f7d-650b4c98d0de" (UID: "d9e7ef62-e7ba-474a-9f7d-650b4c98d0de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:14:44.725430 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.725331 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-kube-api-access-k9xrs" pod "d9e7ef62-e7ba-474a-9f7d-650b4c98d0de" (UID: "d9e7ef62-e7ba-474a-9f7d-650b4c98d0de"). InnerVolumeSpecName "kube-api-access-k9xrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:44.725430 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.725352 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istio-podinfo" pod "d9e7ef62-e7ba-474a-9f7d-650b4c98d0de" (UID: "d9e7ef62-e7ba-474a-9f7d-650b4c98d0de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:14:44.784174 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.784121 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:44.824384 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.824351 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9xrs\" (UniqueName: \"kubernetes.io/projected/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-kube-api-access-k9xrs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:44.824384 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.824384 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:44.824604 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.824397 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:44.824604 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.824411 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:44.824604 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:44.824425 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:45.136885 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:45.136845 4935 generic.go:356] "Generic (PLEG): container finished" podID="d9e7ef62-e7ba-474a-9f7d-650b4c98d0de" containerID="990cc840cc13c6ae300573492bb9de83a243de0b3d2f8e0f59fc113c5efc5203" exitCode=0 Nov 02 22:14:45.137081 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:45.136899 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" Nov 02 22:14:45.137081 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:45.136935 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" event={"ID":"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de","Type":"ContainerDied","Data":"990cc840cc13c6ae300573492bb9de83a243de0b3d2f8e0f59fc113c5efc5203"} Nov 02 22:14:45.137081 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:45.136993 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94" event={"ID":"d9e7ef62-e7ba-474a-9f7d-650b4c98d0de","Type":"ContainerDied","Data":"9765864d4eb116fc3bb93cf896dab9afff13c551c27654322cbcaab46604e2a9"} Nov 02 22:14:45.137081 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:45.137024 4935 scope.go:122] "RemoveContainer" containerID="990cc840cc13c6ae300573492bb9de83a243de0b3d2f8e0f59fc113c5efc5203" Nov 02 22:14:45.149322 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:45.149304 4935 scope.go:122] "RemoveContainer" containerID="aa31aa5acabd4a99eda1990486f703252f925574e10cc265cd0e1840b9735aa2" Nov 02 22:14:45.158741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:45.158704 4935 scope.go:122] "RemoveContainer" containerID="990cc840cc13c6ae300573492bb9de83a243de0b3d2f8e0f59fc113c5efc5203" Nov 02 22:14:45.159050 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:14:45.159028 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"990cc840cc13c6ae300573492bb9de83a243de0b3d2f8e0f59fc113c5efc5203\": container with ID starting with 990cc840cc13c6ae300573492bb9de83a243de0b3d2f8e0f59fc113c5efc5203 not found: ID does not exist" containerID="990cc840cc13c6ae300573492bb9de83a243de0b3d2f8e0f59fc113c5efc5203" Nov 02 22:14:45.159175 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:45.159063 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"990cc840cc13c6ae300573492bb9de83a243de0b3d2f8e0f59fc113c5efc5203"} err="failed to get container status \"990cc840cc13c6ae300573492bb9de83a243de0b3d2f8e0f59fc113c5efc5203\": rpc error: code = NotFound desc = could not find container \"990cc840cc13c6ae300573492bb9de83a243de0b3d2f8e0f59fc113c5efc5203\": container with ID starting with 990cc840cc13c6ae300573492bb9de83a243de0b3d2f8e0f59fc113c5efc5203 not found: ID does not exist" Nov 02 22:14:45.159175 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:45.159088 4935 scope.go:122] "RemoveContainer" containerID="aa31aa5acabd4a99eda1990486f703252f925574e10cc265cd0e1840b9735aa2" Nov 02 22:14:45.159424 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:14:45.159401 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa31aa5acabd4a99eda1990486f703252f925574e10cc265cd0e1840b9735aa2\": container with ID starting with aa31aa5acabd4a99eda1990486f703252f925574e10cc265cd0e1840b9735aa2 not found: ID does not exist" containerID="aa31aa5acabd4a99eda1990486f703252f925574e10cc265cd0e1840b9735aa2" Nov 02 22:14:45.159477 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:45.159437 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa31aa5acabd4a99eda1990486f703252f925574e10cc265cd0e1840b9735aa2"} err="failed to get container status \"aa31aa5acabd4a99eda1990486f703252f925574e10cc265cd0e1840b9735aa2\": rpc error: code = NotFound desc = could not find container \"aa31aa5acabd4a99eda1990486f703252f925574e10cc265cd0e1840b9735aa2\": container with ID starting with aa31aa5acabd4a99eda1990486f703252f925574e10cc265cd0e1840b9735aa2 not found: ID does not exist" Nov 02 22:14:45.291398 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:45.291365 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94"] Nov 02 22:14:45.339777 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:45.339740 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5ddbd0-openshift-default-78bf7b7469-nfj94"] Nov 02 22:14:45.675615 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:45.675571 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:45.780940 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:45.780907 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9e7ef62-e7ba-474a-9f7d-650b4c98d0de" path="/var/lib/kubelet/pods/d9e7ef62-e7ba-474a-9f7d-650b4c98d0de/volumes" Nov 02 22:14:45.936315 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:45.936227 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:46.152118 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:46.152082 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:48.708741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:48.708700 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:48.936052 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:48.936017 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:49.168273 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:49.168232 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:49.728413 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:49.728372 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f"] Nov 02 22:14:49.729195 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:49.729163 4935 cpu_manager.go:414] "containerMap: removing container" podUID="d9e7ef62-e7ba-474a-9f7d-650b4c98d0de" containerName="istio-proxy" Nov 02 22:14:49.729195 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:49.729190 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9e7ef62-e7ba-474a-9f7d-650b4c98d0de" containerName="istio-proxy" Nov 02 22:14:49.729387 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:49.729235 4935 cpu_manager.go:414] "containerMap: removing container" podUID="8bf1a756-7259-404a-aa43-ca017939d6ec" containerName="istio-proxy" Nov 02 22:14:49.729387 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:49.729243 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf1a756-7259-404a-aa43-ca017939d6ec" containerName="istio-proxy" Nov 02 22:14:49.729466 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:49.729414 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf1a756-7259-404a-aa43-ca017939d6ec" containerName="istio-proxy" Nov 02 22:14:49.729466 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:49.729436 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9e7ef62-e7ba-474a-9f7d-650b4c98d0de" containerName="istio-proxy" Nov 02 22:14:49.729466 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:49.729452 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9e7ef62-e7ba-474a-9f7d-650b4c98d0de" containerName="istio-proxy" Nov 02 22:14:49.735064 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:49.735030 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f" Nov 02 22:14:49.783517 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:49.783488 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-d6b384-serving-cert\"" Nov 02 22:14:49.783676 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:49.783524 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f"] Nov 02 22:14:49.787122 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:49.787093 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9b54d14-7be2-4dea-bd32-c801793538bf-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f\" (UID: \"d9b54d14-7be2-4dea-bd32-c801793538bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f" Nov 02 22:14:49.787484 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:49.787161 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9b54d14-7be2-4dea-bd32-c801793538bf-metrics-certs\") pod \"maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f\" (UID: \"d9b54d14-7be2-4dea-bd32-c801793538bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f" Nov 02 22:14:49.787484 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:49.787225 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx56h\" (UniqueName: \"kubernetes.io/projected/d9b54d14-7be2-4dea-bd32-c801793538bf-kube-api-access-bx56h\") pod \"maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f\" (UID: \"d9b54d14-7be2-4dea-bd32-c801793538bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f" Nov 02 22:14:49.787484 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:49.787301 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9b54d14-7be2-4dea-bd32-c801793538bf-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f\" (UID: \"d9b54d14-7be2-4dea-bd32-c801793538bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f" Nov 02 22:14:49.888324 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:49.888294 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9b54d14-7be2-4dea-bd32-c801793538bf-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f\" (UID: \"d9b54d14-7be2-4dea-bd32-c801793538bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f" Nov 02 22:14:49.888504 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:49.888424 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9b54d14-7be2-4dea-bd32-c801793538bf-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f\" (UID: \"d9b54d14-7be2-4dea-bd32-c801793538bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f" Nov 02 22:14:49.888504 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:49.888454 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9b54d14-7be2-4dea-bd32-c801793538bf-metrics-certs\") pod \"maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f\" (UID: \"d9b54d14-7be2-4dea-bd32-c801793538bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f" Nov 02 22:14:49.888504 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:49.888475 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx56h\" (UniqueName: \"kubernetes.io/projected/d9b54d14-7be2-4dea-bd32-c801793538bf-kube-api-access-bx56h\") pod \"maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f\" (UID: \"d9b54d14-7be2-4dea-bd32-c801793538bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f" Nov 02 22:14:49.888624 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:14:49.888566 4935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-serving-cert: secret "maas-api-e2e-extmod-a-d6b384-serving-cert" not found Nov 02 22:14:49.888661 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:14:49.888640 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9b54d14-7be2-4dea-bd32-c801793538bf-maas-api-tls podName:d9b54d14-7be2-4dea-bd32-c801793538bf nodeName:}" failed. No retries permitted until 2026-11-02 22:14:50.388622868 +0000 UTC m=+3982.828197585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d9b54d14-7be2-4dea-bd32-c801793538bf-maas-api-tls") pod "maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f" (UID: "d9b54d14-7be2-4dea-bd32-c801793538bf") : secret "maas-api-e2e-extmod-a-d6b384-serving-cert" not found Nov 02 22:14:49.889065 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:49.889039 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9b54d14-7be2-4dea-bd32-c801793538bf-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f\" (UID: \"d9b54d14-7be2-4dea-bd32-c801793538bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f" Nov 02 22:14:49.890946 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:49.890924 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9b54d14-7be2-4dea-bd32-c801793538bf-metrics-certs\") pod \"maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f\" (UID: \"d9b54d14-7be2-4dea-bd32-c801793538bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f" Nov 02 22:14:49.942344 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:49.942305 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx56h\" (UniqueName: \"kubernetes.io/projected/d9b54d14-7be2-4dea-bd32-c801793538bf-kube-api-access-bx56h\") pod \"maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f\" (UID: \"d9b54d14-7be2-4dea-bd32-c801793538bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f" Nov 02 22:14:50.070624 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:50.070586 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:50.287172 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:50.287118 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:50.395069 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:50.394988 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9b54d14-7be2-4dea-bd32-c801793538bf-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f\" (UID: \"d9b54d14-7be2-4dea-bd32-c801793538bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f" Nov 02 22:14:50.397549 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:50.397523 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9b54d14-7be2-4dea-bd32-c801793538bf-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f\" (UID: \"d9b54d14-7be2-4dea-bd32-c801793538bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f" Nov 02 22:14:50.511411 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:50.511375 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:50.651455 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:50.651374 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f" Nov 02 22:14:51.022559 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:51.022532 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f"] Nov 02 22:14:51.023951 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:14:51.023917 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b54d14_7be2_4dea_bd32_c801793538bf.slice/crio-0d7feec1ef5ed50b38797c2054eeea0522ddf9bc6e2c178af37dcf357f012e0e WatchSource:0}: Error finding container 0d7feec1ef5ed50b38797c2054eeea0522ddf9bc6e2c178af37dcf357f012e0e: Status 404 returned error can't find the container with id 0d7feec1ef5ed50b38797c2054eeea0522ddf9bc6e2c178af37dcf357f012e0e Nov 02 22:14:51.179310 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:51.179274 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f" event={"ID":"d9b54d14-7be2-4dea-bd32-c801793538bf","Type":"ContainerStarted","Data":"0d7feec1ef5ed50b38797c2054eeea0522ddf9bc6e2c178af37dcf357f012e0e"} Nov 02 22:14:52.189020 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:52.188927 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f" event={"ID":"d9b54d14-7be2-4dea-bd32-c801793538bf","Type":"ContainerStarted","Data":"9da3214ea9db2696d3992f9aec2d0053e57414d6d7f3e4dede41a7cf480cfc2c"} Nov 02 22:14:52.189490 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:52.189055 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f" Nov 02 22:14:52.297404 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:52.297344 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f" podStartSLOduration=2.398115649 podStartE2EDuration="3.297329715s" totalImagesPullingTime="899.214066ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:52.295894735 +0000 UTC m=+3984.735469469" watchObservedRunningTime="2026-11-02 22:14:52.297329715 +0000 UTC m=+3984.736904437" Nov 02 22:14:55.638093 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:55.638047 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:58.201194 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:14:58.201167 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f" Nov 02 22:14:58.474870 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:14:58.474779 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:15:00.239109 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:00.239075 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7"] Nov 02 22:15:00.239870 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:00.239845 4935 cpu_manager.go:414] "containerMap: removing container" podUID="d9e7ef62-e7ba-474a-9f7d-650b4c98d0de" containerName="istio-proxy" Nov 02 22:15:00.239870 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:00.239868 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9e7ef62-e7ba-474a-9f7d-650b4c98d0de" containerName="istio-proxy" Nov 02 22:15:00.244535 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:00.244505 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7" Nov 02 22:15:00.321831 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:00.321794 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b2754a-9196-4c85-8def-07eee2023b1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7\" (UID: \"54b2754a-9196-4c85-8def-07eee2023b1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7" Nov 02 22:15:00.322116 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:00.322088 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnb2l\" (UniqueName: \"kubernetes.io/projected/54b2754a-9196-4c85-8def-07eee2023b1f-kube-api-access-vnb2l\") pod \"maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7\" (UID: \"54b2754a-9196-4c85-8def-07eee2023b1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7" Nov 02 22:15:00.328497 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:00.328458 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\"" Nov 02 22:15:00.424362 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:00.424007 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnb2l\" (UniqueName: \"kubernetes.io/projected/54b2754a-9196-4c85-8def-07eee2023b1f-kube-api-access-vnb2l\") pod \"maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7\" (UID: \"54b2754a-9196-4c85-8def-07eee2023b1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7" Nov 02 22:15:00.424682 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:00.424084 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7"] Nov 02 22:15:00.425457 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:00.424707 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b2754a-9196-4c85-8def-07eee2023b1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7\" (UID: \"54b2754a-9196-4c85-8def-07eee2023b1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7" Nov 02 22:15:00.425572 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:00.425546 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b2754a-9196-4c85-8def-07eee2023b1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7\" (UID: \"54b2754a-9196-4c85-8def-07eee2023b1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7" Nov 02 22:15:00.481334 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:00.481296 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnb2l\" (UniqueName: \"kubernetes.io/projected/54b2754a-9196-4c85-8def-07eee2023b1f-kube-api-access-vnb2l\") pod \"maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7\" (UID: \"54b2754a-9196-4c85-8def-07eee2023b1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7" Nov 02 22:15:00.564267 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:00.564234 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7" Nov 02 22:15:00.742654 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:00.742598 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:00.940945 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:00.940853 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7"] Nov 02 22:15:01.257196 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:01.257123 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7" event={"ID":"54b2754a-9196-4c85-8def-07eee2023b1f","Type":"ContainerStarted","Data":"d4b4830aee18a9087f42a38be0822e9ebce5e1753b813813aac2fd2fbabcaab0"} Nov 02 22:15:01.257567 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:01.257225 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7" event={"ID":"54b2754a-9196-4c85-8def-07eee2023b1f","Type":"ContainerStarted","Data":"ae2fb8ce897012c1c3fdb7931c4b16ccf4b789590ff03cdb63f0c4435ec25386"} Nov 02 22:15:01.363206 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:01.363121 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7" podStartSLOduration=1.363104607 podStartE2EDuration="1.363104607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:01.361727664 +0000 UTC m=+3993.801302386" watchObservedRunningTime="2026-11-02 22:15:01.363104607 +0000 UTC m=+3993.802679330" Nov 02 22:15:02.264903 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:02.264867 4935 generic.go:356] "Generic (PLEG): container finished" podID="54b2754a-9196-4c85-8def-07eee2023b1f" containerID="d4b4830aee18a9087f42a38be0822e9ebce5e1753b813813aac2fd2fbabcaab0" exitCode=60 Nov 02 22:15:02.265332 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:02.264950 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7" event={"ID":"54b2754a-9196-4c85-8def-07eee2023b1f","Type":"ContainerDied","Data":"d4b4830aee18a9087f42a38be0822e9ebce5e1753b813813aac2fd2fbabcaab0"} Nov 02 22:15:02.265332 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:02.265225 4935 scope.go:122] "RemoveContainer" containerID="d4b4830aee18a9087f42a38be0822e9ebce5e1753b813813aac2fd2fbabcaab0" Nov 02 22:15:03.273350 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:03.273315 4935 generic.go:356] "Generic (PLEG): container finished" podID="54b2754a-9196-4c85-8def-07eee2023b1f" containerID="33f417e1f61950161cd1b9c8d5df47af16547ce9841e8a70830e40a2ed262e3a" exitCode=60 Nov 02 22:15:03.273780 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:03.273397 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7" event={"ID":"54b2754a-9196-4c85-8def-07eee2023b1f","Type":"ContainerDied","Data":"33f417e1f61950161cd1b9c8d5df47af16547ce9841e8a70830e40a2ed262e3a"} Nov 02 22:15:03.273780 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:03.273463 4935 scope.go:122] "RemoveContainer" containerID="d4b4830aee18a9087f42a38be0822e9ebce5e1753b813813aac2fd2fbabcaab0" Nov 02 22:15:03.273780 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:03.273776 4935 scope.go:122] "RemoveContainer" containerID="33f417e1f61950161cd1b9c8d5df47af16547ce9841e8a70830e40a2ed262e3a" Nov 02 22:15:03.274074 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:15:03.274046 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7_odh-ai-gateway-infra(54b2754a-9196-4c85-8def-07eee2023b1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7" podUID="54b2754a-9196-4c85-8def-07eee2023b1f" Nov 02 22:15:04.282227 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:04.282194 4935 scope.go:122] "RemoveContainer" containerID="33f417e1f61950161cd1b9c8d5df47af16547ce9841e8a70830e40a2ed262e3a" Nov 02 22:15:04.282662 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:15:04.282455 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7_odh-ai-gateway-infra(54b2754a-9196-4c85-8def-07eee2023b1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7" podUID="54b2754a-9196-4c85-8def-07eee2023b1f" Nov 02 22:15:05.722807 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:05.722768 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:10.723065 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:10.722975 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:15.749540 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:15.749500 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:16.769784 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:16.769747 4935 scope.go:122] "RemoveContainer" containerID="33f417e1f61950161cd1b9c8d5df47af16547ce9841e8a70830e40a2ed262e3a" Nov 02 22:15:17.375014 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:17.374930 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7" event={"ID":"54b2754a-9196-4c85-8def-07eee2023b1f","Type":"ContainerStarted","Data":"090a1b3e0b2165b7f6dcc99d02241cb4f4f60b1cae90ddf844e861cab3a3daf6"} Nov 02 22:15:18.382793 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:18.382759 4935 generic.go:356] "Generic (PLEG): container finished" podID="54b2754a-9196-4c85-8def-07eee2023b1f" containerID="090a1b3e0b2165b7f6dcc99d02241cb4f4f60b1cae90ddf844e861cab3a3daf6" exitCode=60 Nov 02 22:15:18.383254 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:18.382829 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7" event={"ID":"54b2754a-9196-4c85-8def-07eee2023b1f","Type":"ContainerDied","Data":"090a1b3e0b2165b7f6dcc99d02241cb4f4f60b1cae90ddf844e861cab3a3daf6"} Nov 02 22:15:18.383254 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:18.382893 4935 scope.go:122] "RemoveContainer" containerID="33f417e1f61950161cd1b9c8d5df47af16547ce9841e8a70830e40a2ed262e3a" Nov 02 22:15:18.383254 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:18.383231 4935 scope.go:122] "RemoveContainer" containerID="090a1b3e0b2165b7f6dcc99d02241cb4f4f60b1cae90ddf844e861cab3a3daf6" Nov 02 22:15:18.383562 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:15:18.383542 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7_odh-ai-gateway-infra(54b2754a-9196-4c85-8def-07eee2023b1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7" podUID="54b2754a-9196-4c85-8def-07eee2023b1f" Nov 02 22:15:18.490263 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:18.490225 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7"] Nov 02 22:15:19.561183 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:19.561153 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7" Nov 02 22:15:19.647584 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:19.647492 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54b2754a-9196-4c85-8def-07eee2023b1f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b2754a-9196-4c85-8def-07eee2023b1f-maas-api-service-ca\") pod \"54b2754a-9196-4c85-8def-07eee2023b1f\" (UID: \"54b2754a-9196-4c85-8def-07eee2023b1f\") " Nov 02 22:15:19.647750 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:19.647587 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b2754a-9196-4c85-8def-07eee2023b1f-kube-api-access-vnb2l\" (UniqueName: \"kubernetes.io/projected/54b2754a-9196-4c85-8def-07eee2023b1f-kube-api-access-vnb2l\") pod \"54b2754a-9196-4c85-8def-07eee2023b1f\" (UID: \"54b2754a-9196-4c85-8def-07eee2023b1f\") " Nov 02 22:15:19.647871 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:19.647847 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54b2754a-9196-4c85-8def-07eee2023b1f-maas-api-service-ca" pod "54b2754a-9196-4c85-8def-07eee2023b1f" (UID: "54b2754a-9196-4c85-8def-07eee2023b1f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:19.648099 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:19.648075 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b2754a-9196-4c85-8def-07eee2023b1f-maas-api-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:19.649657 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:19.649637 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b2754a-9196-4c85-8def-07eee2023b1f-kube-api-access-vnb2l" pod "54b2754a-9196-4c85-8def-07eee2023b1f" (UID: "54b2754a-9196-4c85-8def-07eee2023b1f"). InnerVolumeSpecName "kube-api-access-vnb2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:19.749711 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:19.749673 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnb2l\" (UniqueName: \"kubernetes.io/projected/54b2754a-9196-4c85-8def-07eee2023b1f-kube-api-access-vnb2l\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:19.969591 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:19.969499 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s"] Nov 02 22:15:19.970013 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:19.969957 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s" podUID="8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39" containerName="maas-api" containerID="cri-o://1c2b15fcee9a6e80af0de086e428d25a53da39cd5d09481535c234ea591e4021" gracePeriod=30 Nov 02 22:15:20.153001 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:20.152955 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:20.288943 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:20.288911 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s" Nov 02 22:15:20.362835 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:20.362793 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-metrics-certs\") pod \"8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39\" (UID: \"8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39\") " Nov 02 22:15:20.363036 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:20.363013 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-maas-api-tls\") pod \"8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39\" (UID: \"8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39\") " Nov 02 22:15:20.363109 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:20.363069 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-openshift-service-ca\") pod \"8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39\" (UID: \"8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39\") " Nov 02 22:15:20.363245 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:20.363228 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-kube-api-access-7npr2\" (UniqueName: \"kubernetes.io/projected/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-kube-api-access-7npr2\") pod \"8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39\" (UID: \"8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39\") " Nov 02 22:15:20.363538 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:20.363507 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-openshift-service-ca" pod "8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39" (UID: "8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:20.363855 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:20.363836 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:20.365894 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:20.365866 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-metrics-certs" pod "8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39" (UID: "8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:15:20.366014 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:20.365926 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-kube-api-access-7npr2" pod "8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39" (UID: "8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39"). InnerVolumeSpecName "kube-api-access-7npr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:20.366157 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:20.366104 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-maas-api-tls" pod "8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39" (UID: "8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:15:20.411206 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:20.411160 4935 generic.go:356] "Generic (PLEG): container finished" podID="8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39" containerID="1c2b15fcee9a6e80af0de086e428d25a53da39cd5d09481535c234ea591e4021" exitCode=0 Nov 02 22:15:20.411399 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:20.411216 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s" Nov 02 22:15:20.411399 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:20.411248 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s" event={"ID":"8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39","Type":"ContainerDied","Data":"1c2b15fcee9a6e80af0de086e428d25a53da39cd5d09481535c234ea591e4021"} Nov 02 22:15:20.411399 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:20.411304 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s" event={"ID":"8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39","Type":"ContainerDied","Data":"3ff5f9147e45720c1277f7fa9805b495d83a61fdcf92f67baf0797b7ba9713b6"} Nov 02 22:15:20.411399 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:20.411333 4935 scope.go:122] "RemoveContainer" containerID="1c2b15fcee9a6e80af0de086e428d25a53da39cd5d09481535c234ea591e4021" Nov 02 22:15:20.415938 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:20.415079 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7" event={"ID":"54b2754a-9196-4c85-8def-07eee2023b1f","Type":"ContainerDied","Data":"ae2fb8ce897012c1c3fdb7931c4b16ccf4b789590ff03cdb63f0c4435ec25386"} Nov 02 22:15:20.415938 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:20.415231 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7" Nov 02 22:15:20.432687 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:20.432660 4935 scope.go:122] "RemoveContainer" containerID="1c2b15fcee9a6e80af0de086e428d25a53da39cd5d09481535c234ea591e4021" Nov 02 22:15:20.433094 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:15:20.433062 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c2b15fcee9a6e80af0de086e428d25a53da39cd5d09481535c234ea591e4021\": container with ID starting with 1c2b15fcee9a6e80af0de086e428d25a53da39cd5d09481535c234ea591e4021 not found: ID does not exist" containerID="1c2b15fcee9a6e80af0de086e428d25a53da39cd5d09481535c234ea591e4021" Nov 02 22:15:20.433243 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:20.433110 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c2b15fcee9a6e80af0de086e428d25a53da39cd5d09481535c234ea591e4021"} err="failed to get container status \"1c2b15fcee9a6e80af0de086e428d25a53da39cd5d09481535c234ea591e4021\": rpc error: code = NotFound desc = could not find container \"1c2b15fcee9a6e80af0de086e428d25a53da39cd5d09481535c234ea591e4021\": container with ID starting with 1c2b15fcee9a6e80af0de086e428d25a53da39cd5d09481535c234ea591e4021 not found: ID does not exist" Nov 02 22:15:20.433243 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:20.433149 4935 scope.go:122] "RemoveContainer" containerID="090a1b3e0b2165b7f6dcc99d02241cb4f4f60b1cae90ddf844e861cab3a3daf6" Nov 02 22:15:20.464739 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:20.464705 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:20.464739 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:20.464731 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:20.464739 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:20.464743 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7npr2\" (UniqueName: \"kubernetes.io/projected/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39-kube-api-access-7npr2\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:20.566861 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:20.566825 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s"] Nov 02 22:15:20.574100 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:20.574062 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-065be7-5dcf96c49f-kjz7s"] Nov 02 22:15:20.676373 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:20.676338 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7"] Nov 02 22:15:20.679119 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:20.679095 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6b384-29894295-h84w7"] Nov 02 22:15:21.784651 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:21.784609 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54b2754a-9196-4c85-8def-07eee2023b1f" path="/var/lib/kubelet/pods/54b2754a-9196-4c85-8def-07eee2023b1f/volumes" Nov 02 22:15:21.785938 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:21.785900 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39" path="/var/lib/kubelet/pods/8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39/volumes" Nov 02 22:15:21.940289 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:21.940255 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:24.439965 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:24.439924 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:24.902961 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:24.902925 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:27.237663 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:27.237630 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:27.467732 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:27.467684 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:28.470201 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:15:28.470159 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:15:28.719414 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:28.719385 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:28.927963 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:28.927923 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:31.657489 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:31.657450 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:32.782984 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:32.782951 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:37.802734 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:37.802694 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:39.882843 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:39.882812 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:42.751525 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:42.751489 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:48.214256 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:48.214219 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:53.163318 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:53.163274 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:58.167896 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:15:58.167845 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:58.471262 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:15:58.471184 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:16:02.340847 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:02.340798 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86"] Nov 02 22:16:02.341380 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:02.341076 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86" podUID="9f4cfd18-8663-4bb1-8452-595d606cca1c" containerName="maas-api" containerID="cri-o://54dcb62e0fcb11295a03a65d9c48b4eab216a7916a6f65c3c69575362d5424e2" gracePeriod=30 Nov 02 22:16:02.519762 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:02.519721 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:02.658303 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:02.658274 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86" Nov 02 22:16:02.728097 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:02.727821 4935 generic.go:356] "Generic (PLEG): container finished" podID="9f4cfd18-8663-4bb1-8452-595d606cca1c" containerID="54dcb62e0fcb11295a03a65d9c48b4eab216a7916a6f65c3c69575362d5424e2" exitCode=0 Nov 02 22:16:02.728097 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:02.727876 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86" Nov 02 22:16:02.728097 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:02.727867 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86" event={"ID":"9f4cfd18-8663-4bb1-8452-595d606cca1c","Type":"ContainerDied","Data":"54dcb62e0fcb11295a03a65d9c48b4eab216a7916a6f65c3c69575362d5424e2"} Nov 02 22:16:02.728097 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:02.728158 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86" event={"ID":"9f4cfd18-8663-4bb1-8452-595d606cca1c","Type":"ContainerDied","Data":"9f35cc98029bc4f9c86d10772ee3a11190036162f5c5f8bed0f9271d692c95bc"} Nov 02 22:16:02.728097 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:02.728174 4935 scope.go:122] "RemoveContainer" containerID="54dcb62e0fcb11295a03a65d9c48b4eab216a7916a6f65c3c69575362d5424e2" Nov 02 22:16:02.762155 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:02.758398 4935 scope.go:122] "RemoveContainer" containerID="54dcb62e0fcb11295a03a65d9c48b4eab216a7916a6f65c3c69575362d5424e2" Nov 02 22:16:02.762155 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:16:02.758841 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54dcb62e0fcb11295a03a65d9c48b4eab216a7916a6f65c3c69575362d5424e2\": container with ID starting with 54dcb62e0fcb11295a03a65d9c48b4eab216a7916a6f65c3c69575362d5424e2 not found: ID does not exist" containerID="54dcb62e0fcb11295a03a65d9c48b4eab216a7916a6f65c3c69575362d5424e2" Nov 02 22:16:02.762155 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:02.758886 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54dcb62e0fcb11295a03a65d9c48b4eab216a7916a6f65c3c69575362d5424e2"} err="failed to get container status \"54dcb62e0fcb11295a03a65d9c48b4eab216a7916a6f65c3c69575362d5424e2\": rpc error: code = NotFound desc = could not find container \"54dcb62e0fcb11295a03a65d9c48b4eab216a7916a6f65c3c69575362d5424e2\": container with ID starting with 54dcb62e0fcb11295a03a65d9c48b4eab216a7916a6f65c3c69575362d5424e2 not found: ID does not exist" Nov 02 22:16:02.804006 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:02.803966 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f4cfd18-8663-4bb1-8452-595d606cca1c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f4cfd18-8663-4bb1-8452-595d606cca1c-metrics-certs\") pod \"9f4cfd18-8663-4bb1-8452-595d606cca1c\" (UID: \"9f4cfd18-8663-4bb1-8452-595d606cca1c\") " Nov 02 22:16:02.804212 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:02.804018 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f4cfd18-8663-4bb1-8452-595d606cca1c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f4cfd18-8663-4bb1-8452-595d606cca1c-openshift-service-ca\") pod \"9f4cfd18-8663-4bb1-8452-595d606cca1c\" (UID: \"9f4cfd18-8663-4bb1-8452-595d606cca1c\") " Nov 02 22:16:02.804212 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:02.804074 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f4cfd18-8663-4bb1-8452-595d606cca1c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f4cfd18-8663-4bb1-8452-595d606cca1c-maas-api-tls\") pod \"9f4cfd18-8663-4bb1-8452-595d606cca1c\" (UID: \"9f4cfd18-8663-4bb1-8452-595d606cca1c\") " Nov 02 22:16:02.804349 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:02.804220 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f4cfd18-8663-4bb1-8452-595d606cca1c-kube-api-access-p7fr2\" (UniqueName: \"kubernetes.io/projected/9f4cfd18-8663-4bb1-8452-595d606cca1c-kube-api-access-p7fr2\") pod \"9f4cfd18-8663-4bb1-8452-595d606cca1c\" (UID: \"9f4cfd18-8663-4bb1-8452-595d606cca1c\") " Nov 02 22:16:02.804782 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:02.804746 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f4cfd18-8663-4bb1-8452-595d606cca1c-openshift-service-ca" pod "9f4cfd18-8663-4bb1-8452-595d606cca1c" (UID: "9f4cfd18-8663-4bb1-8452-595d606cca1c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:16:02.806666 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:02.806621 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f4cfd18-8663-4bb1-8452-595d606cca1c-maas-api-tls" pod "9f4cfd18-8663-4bb1-8452-595d606cca1c" (UID: "9f4cfd18-8663-4bb1-8452-595d606cca1c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:16:02.806763 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:02.806702 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4cfd18-8663-4bb1-8452-595d606cca1c-kube-api-access-p7fr2" pod "9f4cfd18-8663-4bb1-8452-595d606cca1c" (UID: "9f4cfd18-8663-4bb1-8452-595d606cca1c"). InnerVolumeSpecName "kube-api-access-p7fr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:16:02.806839 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:02.806813 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f4cfd18-8663-4bb1-8452-595d606cca1c-metrics-certs" pod "9f4cfd18-8663-4bb1-8452-595d606cca1c" (UID: "9f4cfd18-8663-4bb1-8452-595d606cca1c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:16:02.906297 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:02.906180 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7fr2\" (UniqueName: \"kubernetes.io/projected/9f4cfd18-8663-4bb1-8452-595d606cca1c-kube-api-access-p7fr2\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:02.906297 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:02.906218 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f4cfd18-8663-4bb1-8452-595d606cca1c-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:02.906297 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:02.906235 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f4cfd18-8663-4bb1-8452-595d606cca1c-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:02.906297 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:02.906249 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f4cfd18-8663-4bb1-8452-595d606cca1c-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:03.143780 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:03.143742 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86"] Nov 02 22:16:03.189647 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:03.189573 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4585ee-579dffd776-6lp86"] Nov 02 22:16:03.776103 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:03.776072 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f4cfd18-8663-4bb1-8452-595d606cca1c" path="/var/lib/kubelet/pods/9f4cfd18-8663-4bb1-8452-595d606cca1c/volumes" Nov 02 22:16:04.186379 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:04.186285 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:04.408109 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:04.408073 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:04.630970 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:04.630935 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:05.723895 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:05.723858 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:06.136259 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:06.136221 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2"] Nov 02 22:16:06.136469 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:06.136434 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" podUID="cc4fe754-4b3a-43d8-8a8f-12c49c48f936" containerName="istio-proxy" containerID="cri-o://f2aa75c9eb022c0b13d0d3c89213e83204ad8d84a8cc57da3cddd017ad8c3e42" gracePeriod=30 Nov 02 22:16:06.219626 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:06.219593 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:06.810329 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:06.810291 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:08.421141 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:08.421104 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:08.646813 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:08.646779 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:09.639708 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:09.639668 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7"] Nov 02 22:16:09.640240 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:09.640225 4935 cpu_manager.go:414] "containerMap: removing container" podUID="9f4cfd18-8663-4bb1-8452-595d606cca1c" containerName="maas-api" Nov 02 22:16:09.640289 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:09.640242 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f4cfd18-8663-4bb1-8452-595d606cca1c" containerName="maas-api" Nov 02 22:16:09.640289 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:09.640259 4935 cpu_manager.go:414] "containerMap: removing container" podUID="54b2754a-9196-4c85-8def-07eee2023b1f" containerName="cleanup" Nov 02 22:16:09.640289 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:09.640265 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54b2754a-9196-4c85-8def-07eee2023b1f" containerName="cleanup" Nov 02 22:16:09.640289 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:09.640272 4935 cpu_manager.go:414] "containerMap: removing container" podUID="54b2754a-9196-4c85-8def-07eee2023b1f" containerName="cleanup" Nov 02 22:16:09.640289 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:09.640277 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54b2754a-9196-4c85-8def-07eee2023b1f" containerName="cleanup" Nov 02 22:16:09.640449 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:09.640294 4935 cpu_manager.go:414] "containerMap: removing container" podUID="8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39" containerName="maas-api" Nov 02 22:16:09.640449 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:09.640298 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39" containerName="maas-api" Nov 02 22:16:09.640449 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:09.640411 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="54b2754a-9196-4c85-8def-07eee2023b1f" containerName="cleanup" Nov 02 22:16:09.640449 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:09.640420 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f4cfd18-8663-4bb1-8452-595d606cca1c" containerName="maas-api" Nov 02 22:16:09.640449 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:09.640429 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="54b2754a-9196-4c85-8def-07eee2023b1f" containerName="cleanup" Nov 02 22:16:09.640449 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:09.640438 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c7f7bb3-ec3c-444e-80f5-60d71cfcbc39" containerName="maas-api" Nov 02 22:16:09.643863 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:09.643845 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7" Nov 02 22:16:09.691843 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:09.691816 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa69559b-d902-4486-92e3-32af344bf33f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7\" (UID: \"fa69559b-d902-4486-92e3-32af344bf33f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7" Nov 02 22:16:09.691972 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:09.691902 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa69559b-d902-4486-92e3-32af344bf33f-metrics-certs\") pod \"maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7\" (UID: \"fa69559b-d902-4486-92e3-32af344bf33f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7" Nov 02 22:16:09.692053 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:09.692031 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxbdb\" (UniqueName: \"kubernetes.io/projected/fa69559b-d902-4486-92e3-32af344bf33f-kube-api-access-nxbdb\") pod \"maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7\" (UID: \"fa69559b-d902-4486-92e3-32af344bf33f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7" Nov 02 22:16:09.692093 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:09.692080 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa69559b-d902-4486-92e3-32af344bf33f-maas-api-tls\") pod \"maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7\" (UID: \"fa69559b-d902-4486-92e3-32af344bf33f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7" Nov 02 22:16:09.701481 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:09.701463 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-dd28e6-serving-cert\"" Nov 02 22:16:09.747440 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:09.747409 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7"] Nov 02 22:16:09.793416 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:09.793384 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa69559b-d902-4486-92e3-32af344bf33f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7\" (UID: \"fa69559b-d902-4486-92e3-32af344bf33f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7" Nov 02 22:16:09.793597 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:09.793496 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa69559b-d902-4486-92e3-32af344bf33f-metrics-certs\") pod \"maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7\" (UID: \"fa69559b-d902-4486-92e3-32af344bf33f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7" Nov 02 22:16:09.793671 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:09.793623 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxbdb\" (UniqueName: \"kubernetes.io/projected/fa69559b-d902-4486-92e3-32af344bf33f-kube-api-access-nxbdb\") pod \"maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7\" (UID: \"fa69559b-d902-4486-92e3-32af344bf33f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7" Nov 02 22:16:09.793743 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:09.793670 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa69559b-d902-4486-92e3-32af344bf33f-maas-api-tls\") pod \"maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7\" (UID: \"fa69559b-d902-4486-92e3-32af344bf33f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7" Nov 02 22:16:09.793803 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:16:09.793773 4935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-serving-cert: secret "maas-api-e2e-inf-a-dd28e6-serving-cert" not found Nov 02 22:16:09.793861 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:16:09.793846 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa69559b-d902-4486-92e3-32af344bf33f-maas-api-tls podName:fa69559b-d902-4486-92e3-32af344bf33f nodeName:}" failed. No retries permitted until 2026-11-02 22:16:10.293826345 +0000 UTC m=+4062.733401062 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fa69559b-d902-4486-92e3-32af344bf33f-maas-api-tls") pod "maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7" (UID: "fa69559b-d902-4486-92e3-32af344bf33f") : secret "maas-api-e2e-inf-a-dd28e6-serving-cert" not found Nov 02 22:16:09.794244 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:09.794217 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa69559b-d902-4486-92e3-32af344bf33f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7\" (UID: \"fa69559b-d902-4486-92e3-32af344bf33f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7" Nov 02 22:16:09.796316 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:09.796292 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa69559b-d902-4486-92e3-32af344bf33f-metrics-certs\") pod \"maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7\" (UID: \"fa69559b-d902-4486-92e3-32af344bf33f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7" Nov 02 22:16:09.847617 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:09.847571 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxbdb\" (UniqueName: \"kubernetes.io/projected/fa69559b-d902-4486-92e3-32af344bf33f-kube-api-access-nxbdb\") pod \"maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7\" (UID: \"fa69559b-d902-4486-92e3-32af344bf33f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7" Nov 02 22:16:09.962763 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:09.962658 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:10.172541 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:10.172480 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:10.303864 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:10.303828 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa69559b-d902-4486-92e3-32af344bf33f-maas-api-tls\") pod \"maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7\" (UID: \"fa69559b-d902-4486-92e3-32af344bf33f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7" Nov 02 22:16:10.306315 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:10.306291 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa69559b-d902-4486-92e3-32af344bf33f-maas-api-tls\") pod \"maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7\" (UID: \"fa69559b-d902-4486-92e3-32af344bf33f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7" Nov 02 22:16:10.557571 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:10.557485 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7" Nov 02 22:16:10.932777 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:10.931809 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7"] Nov 02 22:16:10.937479 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:16:10.937448 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa69559b_d902_4486_92e3_32af344bf33f.slice/crio-fc1a4686d328ace91b696a661c6352d169399609e1b87b53604fda74d4ba599d WatchSource:0}: Error finding container fc1a4686d328ace91b696a661c6352d169399609e1b87b53604fda74d4ba599d: Status 404 returned error can't find the container with id fc1a4686d328ace91b696a661c6352d169399609e1b87b53604fda74d4ba599d Nov 02 22:16:11.412815 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.412781 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:16:11.518463 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.518431 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-credential-socket\") pod \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " Nov 02 22:16:11.518663 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.518524 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-kube-api-access-p29xr\" (UniqueName: \"kubernetes.io/projected/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-kube-api-access-p29xr\") pod \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " Nov 02 22:16:11.518663 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.518570 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istiod-ca-cert\") pod \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " Nov 02 22:16:11.518663 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.518595 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istio-token\" (UniqueName: \"kubernetes.io/projected/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istio-token\") pod \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " Nov 02 22:16:11.518663 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.518644 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-workload-certs\") pod \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " Nov 02 22:16:11.518890 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.518710 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-workload-socket\") pod \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " Nov 02 22:16:11.518890 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.518750 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istio-podinfo\") pod \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " Nov 02 22:16:11.518890 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.518755 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-credential-socket" pod "cc4fe754-4b3a-43d8-8a8f-12c49c48f936" (UID: "cc4fe754-4b3a-43d8-8a8f-12c49c48f936"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:16:11.518890 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.518862 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istio-data\") pod \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " Nov 02 22:16:11.519089 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.518926 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istio-envoy\") pod \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\" (UID: \"cc4fe754-4b3a-43d8-8a8f-12c49c48f936\") " Nov 02 22:16:11.519089 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.518991 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-workload-certs" pod "cc4fe754-4b3a-43d8-8a8f-12c49c48f936" (UID: "cc4fe754-4b3a-43d8-8a8f-12c49c48f936"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:16:11.519089 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.519004 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istiod-ca-cert" pod "cc4fe754-4b3a-43d8-8a8f-12c49c48f936" (UID: "cc4fe754-4b3a-43d8-8a8f-12c49c48f936"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:16:11.519273 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.519101 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-workload-socket" pod "cc4fe754-4b3a-43d8-8a8f-12c49c48f936" (UID: "cc4fe754-4b3a-43d8-8a8f-12c49c48f936"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:16:11.519634 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.519444 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:11.519634 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.519468 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:11.519634 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.519483 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:11.519634 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.519498 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:11.519634 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.519599 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istio-data" pod "cc4fe754-4b3a-43d8-8a8f-12c49c48f936" (UID: "cc4fe754-4b3a-43d8-8a8f-12c49c48f936"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:16:11.521226 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.521204 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istio-envoy" pod "cc4fe754-4b3a-43d8-8a8f-12c49c48f936" (UID: "cc4fe754-4b3a-43d8-8a8f-12c49c48f936"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:16:11.521433 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.521413 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istio-token" pod "cc4fe754-4b3a-43d8-8a8f-12c49c48f936" (UID: "cc4fe754-4b3a-43d8-8a8f-12c49c48f936"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:16:11.521670 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.521647 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istio-podinfo" pod "cc4fe754-4b3a-43d8-8a8f-12c49c48f936" (UID: "cc4fe754-4b3a-43d8-8a8f-12c49c48f936"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:16:11.521799 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.521779 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-kube-api-access-p29xr" pod "cc4fe754-4b3a-43d8-8a8f-12c49c48f936" (UID: "cc4fe754-4b3a-43d8-8a8f-12c49c48f936"). InnerVolumeSpecName "kube-api-access-p29xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:16:11.620390 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.620350 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:11.620390 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.620383 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:11.620390 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.620393 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:11.620390 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.620402 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:11.620751 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.620411 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p29xr\" (UniqueName: \"kubernetes.io/projected/cc4fe754-4b3a-43d8-8a8f-12c49c48f936-kube-api-access-p29xr\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:11.790876 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.790843 4935 generic.go:356] "Generic (PLEG): container finished" podID="cc4fe754-4b3a-43d8-8a8f-12c49c48f936" containerID="f2aa75c9eb022c0b13d0d3c89213e83204ad8d84a8cc57da3cddd017ad8c3e42" exitCode=0 Nov 02 22:16:11.790991 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.790897 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" Nov 02 22:16:11.790991 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.790918 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" event={"ID":"cc4fe754-4b3a-43d8-8a8f-12c49c48f936","Type":"ContainerDied","Data":"f2aa75c9eb022c0b13d0d3c89213e83204ad8d84a8cc57da3cddd017ad8c3e42"} Nov 02 22:16:11.790991 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.790972 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2" event={"ID":"cc4fe754-4b3a-43d8-8a8f-12c49c48f936","Type":"ContainerDied","Data":"6f2bfffcf8126588e70341fbe68f13bc1217cf2a29c598a8ef429945d741d37d"} Nov 02 22:16:11.791145 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.790997 4935 scope.go:122] "RemoveContainer" containerID="f2aa75c9eb022c0b13d0d3c89213e83204ad8d84a8cc57da3cddd017ad8c3e42" Nov 02 22:16:11.792314 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.792292 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7" event={"ID":"fa69559b-d902-4486-92e3-32af344bf33f","Type":"ContainerStarted","Data":"fc1a4686d328ace91b696a661c6352d169399609e1b87b53604fda74d4ba599d"} Nov 02 22:16:11.802226 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.802209 4935 scope.go:122] "RemoveContainer" containerID="30b8a87e6aec53b686771ab450689f797e97fec03614d000ce03ea4e8f1f9352" Nov 02 22:16:11.811289 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.811274 4935 scope.go:122] "RemoveContainer" containerID="f2aa75c9eb022c0b13d0d3c89213e83204ad8d84a8cc57da3cddd017ad8c3e42" Nov 02 22:16:11.811607 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:16:11.811586 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2aa75c9eb022c0b13d0d3c89213e83204ad8d84a8cc57da3cddd017ad8c3e42\": container with ID starting with f2aa75c9eb022c0b13d0d3c89213e83204ad8d84a8cc57da3cddd017ad8c3e42 not found: ID does not exist" containerID="f2aa75c9eb022c0b13d0d3c89213e83204ad8d84a8cc57da3cddd017ad8c3e42" Nov 02 22:16:11.811660 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.811614 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2aa75c9eb022c0b13d0d3c89213e83204ad8d84a8cc57da3cddd017ad8c3e42"} err="failed to get container status \"f2aa75c9eb022c0b13d0d3c89213e83204ad8d84a8cc57da3cddd017ad8c3e42\": rpc error: code = NotFound desc = could not find container \"f2aa75c9eb022c0b13d0d3c89213e83204ad8d84a8cc57da3cddd017ad8c3e42\": container with ID starting with f2aa75c9eb022c0b13d0d3c89213e83204ad8d84a8cc57da3cddd017ad8c3e42 not found: ID does not exist" Nov 02 22:16:11.811660 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.811629 4935 scope.go:122] "RemoveContainer" containerID="30b8a87e6aec53b686771ab450689f797e97fec03614d000ce03ea4e8f1f9352" Nov 02 22:16:11.811904 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:16:11.811878 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30b8a87e6aec53b686771ab450689f797e97fec03614d000ce03ea4e8f1f9352\": container with ID starting with 30b8a87e6aec53b686771ab450689f797e97fec03614d000ce03ea4e8f1f9352 not found: ID does not exist" containerID="30b8a87e6aec53b686771ab450689f797e97fec03614d000ce03ea4e8f1f9352" Nov 02 22:16:11.811950 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.811901 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30b8a87e6aec53b686771ab450689f797e97fec03614d000ce03ea4e8f1f9352"} err="failed to get container status \"30b8a87e6aec53b686771ab450689f797e97fec03614d000ce03ea4e8f1f9352\": rpc error: code = NotFound desc = could not find container \"30b8a87e6aec53b686771ab450689f797e97fec03614d000ce03ea4e8f1f9352\": container with ID starting with 30b8a87e6aec53b686771ab450689f797e97fec03614d000ce03ea4e8f1f9352 not found: ID does not exist" Nov 02 22:16:11.949809 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.949775 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2"] Nov 02 22:16:11.995331 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:11.995302 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4585ee-openshift-default-6db5694d99-6k8k2"] Nov 02 22:16:12.722372 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:12.722338 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:12.801336 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:12.801301 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7" event={"ID":"fa69559b-d902-4486-92e3-32af344bf33f","Type":"ContainerStarted","Data":"68edf1ffcc942c2e08ff11cf95c654c336640bfb0aa1efb9bd4db1681241ac6d"} Nov 02 22:16:12.801547 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:12.801373 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7" Nov 02 22:16:12.908257 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:12.908207 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7" podStartSLOduration=2.882930964 podStartE2EDuration="3.908193029s" totalImagesPullingTime="1.025262065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:16:12.906666182 +0000 UTC m=+4065.346240903" watchObservedRunningTime="2026-11-02 22:16:12.908193029 +0000 UTC m=+4065.347767750" Nov 02 22:16:13.436973 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:13.436935 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:13.643635 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:13.643597 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:13.807840 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:13.807781 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc4fe754-4b3a-43d8-8a8f-12c49c48f936" path="/var/lib/kubelet/pods/cc4fe754-4b3a-43d8-8a8f-12c49c48f936/volumes" Nov 02 22:16:15.510842 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:15.510810 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:18.155809 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:18.155765 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:18.825801 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:18.825767 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7" Nov 02 22:16:20.546101 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:20.546065 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:21.982844 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:21.982806 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:24.795357 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:24.795316 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:25.508698 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:25.508663 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:27.771331 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:27.771286 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:28.010922 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:28.010890 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:28.470366 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:16:28.470323 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:16:30.453577 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:30.453549 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:30.657704 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:30.657673 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:33.492397 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:33.492361 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:35.542953 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:35.542896 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:40.279841 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:40.279754 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:40.479280 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:40.479245 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:40.681824 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:40.681742 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:45.604723 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:45.604686 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:46.919395 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:46.919363 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:50.618585 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:50.618552 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:52.900355 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:52.900321 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:55.577052 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:55.577009 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:56.676481 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:16:56.676443 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:58.470048 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:16:58.470012 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:17:00.574821 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:00.574785 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:05.620805 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:05.620768 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:10.593750 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:10.593682 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:15.555800 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:15.555763 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:20.517768 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:20.517729 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:24.703881 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:24.703847 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh"] Nov 02 22:17:24.704351 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:24.704042 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh" podUID="94192fa7-1b0f-4384-9207-b0caf7e4eac7" containerName="maas-api" containerID="cri-o://f8c60b3b1711ad7fbaf20ba8c13a6a8e48fe074652c6c61d5324d87cb0177533" gracePeriod=30 Nov 02 22:17:24.876910 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:24.876628 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:25.013346 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:25.013319 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh" Nov 02 22:17:25.173825 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:25.170821 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94192fa7-1b0f-4384-9207-b0caf7e4eac7-kube-api-access-zwlsk\" (UniqueName: \"kubernetes.io/projected/94192fa7-1b0f-4384-9207-b0caf7e4eac7-kube-api-access-zwlsk\") pod \"94192fa7-1b0f-4384-9207-b0caf7e4eac7\" (UID: \"94192fa7-1b0f-4384-9207-b0caf7e4eac7\") " Nov 02 22:17:25.173825 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:25.170936 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94192fa7-1b0f-4384-9207-b0caf7e4eac7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/94192fa7-1b0f-4384-9207-b0caf7e4eac7-metrics-certs\") pod \"94192fa7-1b0f-4384-9207-b0caf7e4eac7\" (UID: \"94192fa7-1b0f-4384-9207-b0caf7e4eac7\") " Nov 02 22:17:25.173825 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:25.171022 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94192fa7-1b0f-4384-9207-b0caf7e4eac7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94192fa7-1b0f-4384-9207-b0caf7e4eac7-maas-api-tls\") pod \"94192fa7-1b0f-4384-9207-b0caf7e4eac7\" (UID: \"94192fa7-1b0f-4384-9207-b0caf7e4eac7\") " Nov 02 22:17:25.173825 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:25.171151 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94192fa7-1b0f-4384-9207-b0caf7e4eac7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94192fa7-1b0f-4384-9207-b0caf7e4eac7-openshift-service-ca\") pod \"94192fa7-1b0f-4384-9207-b0caf7e4eac7\" (UID: \"94192fa7-1b0f-4384-9207-b0caf7e4eac7\") " Nov 02 22:17:25.173825 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:25.171694 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94192fa7-1b0f-4384-9207-b0caf7e4eac7-openshift-service-ca" pod "94192fa7-1b0f-4384-9207-b0caf7e4eac7" (UID: "94192fa7-1b0f-4384-9207-b0caf7e4eac7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:17:25.174152 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:25.173855 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94192fa7-1b0f-4384-9207-b0caf7e4eac7-maas-api-tls" pod "94192fa7-1b0f-4384-9207-b0caf7e4eac7" (UID: "94192fa7-1b0f-4384-9207-b0caf7e4eac7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:17:25.176552 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:25.176521 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94192fa7-1b0f-4384-9207-b0caf7e4eac7-metrics-certs" pod "94192fa7-1b0f-4384-9207-b0caf7e4eac7" (UID: "94192fa7-1b0f-4384-9207-b0caf7e4eac7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:17:25.178248 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:25.178223 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94192fa7-1b0f-4384-9207-b0caf7e4eac7-kube-api-access-zwlsk" pod "94192fa7-1b0f-4384-9207-b0caf7e4eac7" (UID: "94192fa7-1b0f-4384-9207-b0caf7e4eac7"). InnerVolumeSpecName "kube-api-access-zwlsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:17:25.272830 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:25.272788 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwlsk\" (UniqueName: \"kubernetes.io/projected/94192fa7-1b0f-4384-9207-b0caf7e4eac7-kube-api-access-zwlsk\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:17:25.272830 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:25.272829 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94192fa7-1b0f-4384-9207-b0caf7e4eac7-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:17:25.273079 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:25.272847 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94192fa7-1b0f-4384-9207-b0caf7e4eac7-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:17:25.273079 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:25.272861 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94192fa7-1b0f-4384-9207-b0caf7e4eac7-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:17:25.326123 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:25.326086 4935 generic.go:356] "Generic (PLEG): container finished" podID="94192fa7-1b0f-4384-9207-b0caf7e4eac7" containerID="f8c60b3b1711ad7fbaf20ba8c13a6a8e48fe074652c6c61d5324d87cb0177533" exitCode=0 Nov 02 22:17:25.326294 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:25.326159 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh" Nov 02 22:17:25.326294 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:25.326174 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh" event={"ID":"94192fa7-1b0f-4384-9207-b0caf7e4eac7","Type":"ContainerDied","Data":"f8c60b3b1711ad7fbaf20ba8c13a6a8e48fe074652c6c61d5324d87cb0177533"} Nov 02 22:17:25.326294 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:25.326237 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh" event={"ID":"94192fa7-1b0f-4384-9207-b0caf7e4eac7","Type":"ContainerDied","Data":"38e54e5e17d93146d08f8d9ced32ffdfec84d776d0a80d48645f5485f8857315"} Nov 02 22:17:25.326294 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:25.326260 4935 scope.go:122] "RemoveContainer" containerID="f8c60b3b1711ad7fbaf20ba8c13a6a8e48fe074652c6c61d5324d87cb0177533" Nov 02 22:17:25.337144 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:25.337118 4935 scope.go:122] "RemoveContainer" containerID="f8c60b3b1711ad7fbaf20ba8c13a6a8e48fe074652c6c61d5324d87cb0177533" Nov 02 22:17:25.337464 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:17:25.337444 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8c60b3b1711ad7fbaf20ba8c13a6a8e48fe074652c6c61d5324d87cb0177533\": container with ID starting with f8c60b3b1711ad7fbaf20ba8c13a6a8e48fe074652c6c61d5324d87cb0177533 not found: ID does not exist" containerID="f8c60b3b1711ad7fbaf20ba8c13a6a8e48fe074652c6c61d5324d87cb0177533" Nov 02 22:17:25.337568 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:25.337470 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c60b3b1711ad7fbaf20ba8c13a6a8e48fe074652c6c61d5324d87cb0177533"} err="failed to get container status \"f8c60b3b1711ad7fbaf20ba8c13a6a8e48fe074652c6c61d5324d87cb0177533\": rpc error: code = NotFound desc = could not find container \"f8c60b3b1711ad7fbaf20ba8c13a6a8e48fe074652c6c61d5324d87cb0177533\": container with ID starting with f8c60b3b1711ad7fbaf20ba8c13a6a8e48fe074652c6c61d5324d87cb0177533 not found: ID does not exist" Nov 02 22:17:25.478598 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:25.478566 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh"] Nov 02 22:17:25.524845 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:25.524768 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72f279-5784755887-vlpwh"] Nov 02 22:17:25.777035 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:25.776955 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94192fa7-1b0f-4384-9207-b0caf7e4eac7" path="/var/lib/kubelet/pods/94192fa7-1b0f-4384-9207-b0caf7e4eac7/volumes" Nov 02 22:17:26.675303 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:26.675265 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:28.469848 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:17:28.469804 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:17:31.619839 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:31.619793 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:32.347717 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:32.347670 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:32.552433 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:32.552395 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:33.874106 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:33.874070 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:34.386023 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:34.385990 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:36.684223 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:36.684184 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:36.900598 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:36.900560 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:37.108450 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:37.108410 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:37.910079 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:37.910029 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:38.124971 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:38.124926 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:39.396650 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:39.396612 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:39.803101 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:39.803065 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:40.465809 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:40.465766 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:40.684095 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:40.684055 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:40.919973 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:40.919940 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:41.782404 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:41.782370 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f"] Nov 02 22:17:41.782817 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:41.782609 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f" podUID="d9b54d14-7be2-4dea-bd32-c801793538bf" containerName="maas-api" containerID="cri-o://9da3214ea9db2696d3992f9aec2d0053e57414d6d7f3e4dede41a7cf480cfc2c" gracePeriod=30 Nov 02 22:17:41.899607 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:41.899538 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:42.100468 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:42.100444 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f" Nov 02 22:17:42.202387 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:42.202353 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9b54d14-7be2-4dea-bd32-c801793538bf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9b54d14-7be2-4dea-bd32-c801793538bf-maas-api-tls\") pod \"d9b54d14-7be2-4dea-bd32-c801793538bf\" (UID: \"d9b54d14-7be2-4dea-bd32-c801793538bf\") " Nov 02 22:17:42.202577 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:42.202441 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9b54d14-7be2-4dea-bd32-c801793538bf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9b54d14-7be2-4dea-bd32-c801793538bf-metrics-certs\") pod \"d9b54d14-7be2-4dea-bd32-c801793538bf\" (UID: \"d9b54d14-7be2-4dea-bd32-c801793538bf\") " Nov 02 22:17:42.202577 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:42.202500 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b54d14-7be2-4dea-bd32-c801793538bf-kube-api-access-bx56h\" (UniqueName: \"kubernetes.io/projected/d9b54d14-7be2-4dea-bd32-c801793538bf-kube-api-access-bx56h\") pod \"d9b54d14-7be2-4dea-bd32-c801793538bf\" (UID: \"d9b54d14-7be2-4dea-bd32-c801793538bf\") " Nov 02 22:17:42.202577 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:42.202546 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9b54d14-7be2-4dea-bd32-c801793538bf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9b54d14-7be2-4dea-bd32-c801793538bf-openshift-service-ca\") pod \"d9b54d14-7be2-4dea-bd32-c801793538bf\" (UID: \"d9b54d14-7be2-4dea-bd32-c801793538bf\") " Nov 02 22:17:42.203033 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:42.202996 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9b54d14-7be2-4dea-bd32-c801793538bf-openshift-service-ca" pod "d9b54d14-7be2-4dea-bd32-c801793538bf" (UID: "d9b54d14-7be2-4dea-bd32-c801793538bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:17:42.204826 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:42.204796 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9b54d14-7be2-4dea-bd32-c801793538bf-maas-api-tls" pod "d9b54d14-7be2-4dea-bd32-c801793538bf" (UID: "d9b54d14-7be2-4dea-bd32-c801793538bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:17:42.205043 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:42.205017 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9b54d14-7be2-4dea-bd32-c801793538bf-metrics-certs" pod "d9b54d14-7be2-4dea-bd32-c801793538bf" (UID: "d9b54d14-7be2-4dea-bd32-c801793538bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:17:42.205184 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:42.205163 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b54d14-7be2-4dea-bd32-c801793538bf-kube-api-access-bx56h" pod "d9b54d14-7be2-4dea-bd32-c801793538bf" (UID: "d9b54d14-7be2-4dea-bd32-c801793538bf"). InnerVolumeSpecName "kube-api-access-bx56h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:17:42.304060 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:42.304022 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9b54d14-7be2-4dea-bd32-c801793538bf-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:17:42.304060 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:42.304054 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9b54d14-7be2-4dea-bd32-c801793538bf-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:17:42.304060 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:42.304066 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx56h\" (UniqueName: \"kubernetes.io/projected/d9b54d14-7be2-4dea-bd32-c801793538bf-kube-api-access-bx56h\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:17:42.304306 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:42.304077 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9b54d14-7be2-4dea-bd32-c801793538bf-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:17:42.440638 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:42.440552 4935 generic.go:356] "Generic (PLEG): container finished" podID="d9b54d14-7be2-4dea-bd32-c801793538bf" containerID="9da3214ea9db2696d3992f9aec2d0053e57414d6d7f3e4dede41a7cf480cfc2c" exitCode=0 Nov 02 22:17:42.440795 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:42.440634 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f" Nov 02 22:17:42.440795 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:42.440641 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f" event={"ID":"d9b54d14-7be2-4dea-bd32-c801793538bf","Type":"ContainerDied","Data":"9da3214ea9db2696d3992f9aec2d0053e57414d6d7f3e4dede41a7cf480cfc2c"} Nov 02 22:17:42.440795 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:42.440751 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f" event={"ID":"d9b54d14-7be2-4dea-bd32-c801793538bf","Type":"ContainerDied","Data":"0d7feec1ef5ed50b38797c2054eeea0522ddf9bc6e2c178af37dcf357f012e0e"} Nov 02 22:17:42.440795 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:42.440779 4935 scope.go:122] "RemoveContainer" containerID="9da3214ea9db2696d3992f9aec2d0053e57414d6d7f3e4dede41a7cf480cfc2c" Nov 02 22:17:42.451475 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:42.451457 4935 scope.go:122] "RemoveContainer" containerID="9da3214ea9db2696d3992f9aec2d0053e57414d6d7f3e4dede41a7cf480cfc2c" Nov 02 22:17:42.451806 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:17:42.451778 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9da3214ea9db2696d3992f9aec2d0053e57414d6d7f3e4dede41a7cf480cfc2c\": container with ID starting with 9da3214ea9db2696d3992f9aec2d0053e57414d6d7f3e4dede41a7cf480cfc2c not found: ID does not exist" containerID="9da3214ea9db2696d3992f9aec2d0053e57414d6d7f3e4dede41a7cf480cfc2c" Nov 02 22:17:42.451922 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:42.451812 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9da3214ea9db2696d3992f9aec2d0053e57414d6d7f3e4dede41a7cf480cfc2c"} err="failed to get container status \"9da3214ea9db2696d3992f9aec2d0053e57414d6d7f3e4dede41a7cf480cfc2c\": rpc error: code = NotFound desc = could not find container \"9da3214ea9db2696d3992f9aec2d0053e57414d6d7f3e4dede41a7cf480cfc2c\": container with ID starting with 9da3214ea9db2696d3992f9aec2d0053e57414d6d7f3e4dede41a7cf480cfc2c not found: ID does not exist" Nov 02 22:17:42.593567 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:42.593529 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f"] Nov 02 22:17:42.641750 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:42.641716 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6b384-557fd49cdc-4qx5f"] Nov 02 22:17:43.776196 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:43.776125 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9b54d14-7be2-4dea-bd32-c801793538bf" path="/var/lib/kubelet/pods/d9b54d14-7be2-4dea-bd32-c801793538bf/volumes" Nov 02 22:17:43.834858 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:43.834819 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:44.993660 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:44.993629 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:45.212310 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:45.212270 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:47.613939 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:47.613910 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:47.850546 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:47.850514 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:48.132309 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:48.132272 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:48.344022 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:48.343985 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:49.348742 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:49.348708 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:49.573601 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:49.573563 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:49.817904 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:49.817871 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:50.776921 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:50.776887 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:51.901850 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:51.901814 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:52.481882 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:52.481839 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:52.817920 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:52.817871 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:54.928317 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:54.928277 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:55.158226 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:55.158187 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:56.858757 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:56.858717 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:57.095769 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:57.095733 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:57.616396 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:17:57.616365 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:58.469873 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:17:58.469828 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:18:00.245348 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:18:00.245300 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:00.458831 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:18:00.458794 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:05.527347 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:18:05.527312 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:05.796829 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:18:05.796745 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:10.479819 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:18:10.479734 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:12.056841 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:18:12.056803 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:15.524905 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:18:15.524865 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:17.365047 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:18:17.365004 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:17.598704 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:18:17.598656 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:18.224800 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:18:18.224766 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:20.727180 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:18:20.727122 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:25.720865 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:18:25.720789 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:28.469996 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:18:28.469950 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:18:28.549758 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:18:28.549723 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 22:18:28.561021 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:18:28.560994 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 22:18:28.584220 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:18:28.584190 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 22:18:28.592900 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:18:28.592877 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 22:18:30.731774 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:18:30.731726 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:35.675290 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:18:35.675253 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:35.929595 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:18:35.929507 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:36.126213 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:18:36.126121 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:40.734464 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:18:40.734430 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:40.965190 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:18:40.965151 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:44.433594 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:18:44.433554 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:45.616811 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:18:45.616775 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:50.211415 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:18:50.211287 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:50.626742 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:18:50.626700 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:55.581927 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:18:55.581889 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:58.471706 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:18:58.471654 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:19:00.161403 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.161360 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt"] Nov 02 22:19:00.161915 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.161863 4935 cpu_manager.go:414] "containerMap: removing container" podUID="d9b54d14-7be2-4dea-bd32-c801793538bf" containerName="maas-api" Nov 02 22:19:00.161915 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.161876 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9b54d14-7be2-4dea-bd32-c801793538bf" containerName="maas-api" Nov 02 22:19:00.161915 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.161894 4935 cpu_manager.go:414] "containerMap: removing container" podUID="94192fa7-1b0f-4384-9207-b0caf7e4eac7" containerName="maas-api" Nov 02 22:19:00.161915 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.161899 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94192fa7-1b0f-4384-9207-b0caf7e4eac7" containerName="maas-api" Nov 02 22:19:00.161915 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.161905 4935 cpu_manager.go:414] "containerMap: removing container" podUID="54b2754a-9196-4c85-8def-07eee2023b1f" containerName="cleanup" Nov 02 22:19:00.161915 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.161910 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54b2754a-9196-4c85-8def-07eee2023b1f" containerName="cleanup" Nov 02 22:19:00.162270 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.161931 4935 cpu_manager.go:414] "containerMap: removing container" podUID="cc4fe754-4b3a-43d8-8a8f-12c49c48f936" containerName="istio-proxy" Nov 02 22:19:00.162270 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.161936 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc4fe754-4b3a-43d8-8a8f-12c49c48f936" containerName="istio-proxy" Nov 02 22:19:00.162270 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.161944 4935 cpu_manager.go:414] "containerMap: removing container" podUID="cc4fe754-4b3a-43d8-8a8f-12c49c48f936" containerName="istio-proxy" Nov 02 22:19:00.162270 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.161949 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc4fe754-4b3a-43d8-8a8f-12c49c48f936" containerName="istio-proxy" Nov 02 22:19:00.162270 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.162030 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9b54d14-7be2-4dea-bd32-c801793538bf" containerName="maas-api" Nov 02 22:19:00.162270 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.162042 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="94192fa7-1b0f-4384-9207-b0caf7e4eac7" containerName="maas-api" Nov 02 22:19:00.162270 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.162048 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="54b2754a-9196-4c85-8def-07eee2023b1f" containerName="cleanup" Nov 02 22:19:00.162270 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.162057 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc4fe754-4b3a-43d8-8a8f-12c49c48f936" containerName="istio-proxy" Nov 02 22:19:00.162270 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.162064 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc4fe754-4b3a-43d8-8a8f-12c49c48f936" containerName="istio-proxy" Nov 02 22:19:00.165063 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.164990 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt" Nov 02 22:19:00.218700 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.218667 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt"] Nov 02 22:19:00.249264 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.249224 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-cf2a53-serving-cert\"" Nov 02 22:19:00.262951 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.262909 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-maas-api-tls\") pod \"maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt\" (UID: \"7e3aac8c-f5ba-47d1-aae1-5baf948384b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt" Nov 02 22:19:00.263105 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.262963 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-metrics-certs\") pod \"maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt\" (UID: \"7e3aac8c-f5ba-47d1-aae1-5baf948384b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt" Nov 02 22:19:00.263204 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.263177 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwsdv\" (UniqueName: \"kubernetes.io/projected/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-kube-api-access-cwsdv\") pod \"maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt\" (UID: \"7e3aac8c-f5ba-47d1-aae1-5baf948384b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt" Nov 02 22:19:00.263365 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.263346 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt\" (UID: \"7e3aac8c-f5ba-47d1-aae1-5baf948384b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt" Nov 02 22:19:00.365118 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.365076 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt\" (UID: \"7e3aac8c-f5ba-47d1-aae1-5baf948384b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt" Nov 02 22:19:00.365319 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.365184 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-maas-api-tls\") pod \"maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt\" (UID: \"7e3aac8c-f5ba-47d1-aae1-5baf948384b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt" Nov 02 22:19:00.365319 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.365233 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-metrics-certs\") pod \"maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt\" (UID: \"7e3aac8c-f5ba-47d1-aae1-5baf948384b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt" Nov 02 22:19:00.365441 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.365332 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwsdv\" (UniqueName: \"kubernetes.io/projected/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-kube-api-access-cwsdv\") pod \"maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt\" (UID: \"7e3aac8c-f5ba-47d1-aae1-5baf948384b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt" Nov 02 22:19:00.365441 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:19:00.365336 4935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-serving-cert: secret "maas-api-e2e-inf-b-cf2a53-serving-cert" not found Nov 02 22:19:00.365441 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:19:00.365410 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-maas-api-tls podName:7e3aac8c-f5ba-47d1-aae1-5baf948384b5 nodeName:}" failed. No retries permitted until 2026-11-02 22:19:00.865389336 +0000 UTC m=+4233.304964041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-maas-api-tls") pod "maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt" (UID: "7e3aac8c-f5ba-47d1-aae1-5baf948384b5") : secret "maas-api-e2e-inf-b-cf2a53-serving-cert" not found Nov 02 22:19:00.365850 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.365824 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt\" (UID: \"7e3aac8c-f5ba-47d1-aae1-5baf948384b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt" Nov 02 22:19:00.367987 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.367957 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-metrics-certs\") pod \"maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt\" (UID: \"7e3aac8c-f5ba-47d1-aae1-5baf948384b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt" Nov 02 22:19:00.419061 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.418991 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwsdv\" (UniqueName: \"kubernetes.io/projected/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-kube-api-access-cwsdv\") pod \"maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt\" (UID: \"7e3aac8c-f5ba-47d1-aae1-5baf948384b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt" Nov 02 22:19:00.437126 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.437078 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:00.644744 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.644686 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:00.840170 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.840114 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:00.871112 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.871073 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-maas-api-tls\") pod \"maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt\" (UID: \"7e3aac8c-f5ba-47d1-aae1-5baf948384b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt" Nov 02 22:19:00.873977 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:00.873951 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-maas-api-tls\") pod \"maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt\" (UID: \"7e3aac8c-f5ba-47d1-aae1-5baf948384b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt" Nov 02 22:19:01.079803 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:01.079754 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt" Nov 02 22:19:01.448570 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:01.448542 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt"] Nov 02 22:19:01.450300 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:19:01.450268 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3aac8c_f5ba_47d1_aae1_5baf948384b5.slice/crio-f70d196da05aa23139a0573e02afd88a2f1846846cbe0684c8d1c48ab2ecb007 WatchSource:0}: Error finding container f70d196da05aa23139a0573e02afd88a2f1846846cbe0684c8d1c48ab2ecb007: Status 404 returned error can't find the container with id f70d196da05aa23139a0573e02afd88a2f1846846cbe0684c8d1c48ab2ecb007 Nov 02 22:19:01.451962 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:01.451942 4935 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:19:01.995702 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:01.995662 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt" event={"ID":"7e3aac8c-f5ba-47d1-aae1-5baf948384b5","Type":"ContainerStarted","Data":"f70d196da05aa23139a0573e02afd88a2f1846846cbe0684c8d1c48ab2ecb007"} Nov 02 22:19:03.004561 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:03.004469 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt" event={"ID":"7e3aac8c-f5ba-47d1-aae1-5baf948384b5","Type":"ContainerStarted","Data":"2cf98ebbadfc391dc269818c7470cdea134ef92a44869d28d5814c5a5f7c9b23"} Nov 02 22:19:03.004954 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:03.004576 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt" Nov 02 22:19:03.114054 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:03.114004 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt" podStartSLOduration=2.047476656 podStartE2EDuration="3.113990195s" totalImagesPullingTime="1.066513539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:19:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:19:03.111109731 +0000 UTC m=+4235.550684453" watchObservedRunningTime="2026-11-02 22:19:03.113990195 +0000 UTC m=+4235.553564959" Nov 02 22:19:05.651774 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:05.651743 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:09.015519 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:09.015485 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt" Nov 02 22:19:10.720290 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:10.720252 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:15.684823 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:15.684785 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:15.887726 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:15.887694 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:20.663831 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:20.663744 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:20.916329 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:20.916225 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:25.933900 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:25.933868 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:28.473176 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:19:28.473112 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:19:30.933764 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:30.933723 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:32.358266 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:32.358232 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:35.933150 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:35.932751 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:38.119641 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:38.119538 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:40.888678 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:40.888638 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:41.330076 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:41.330039 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:42.183177 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:42.183122 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:42.394997 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:42.394961 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:42.607884 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:42.607850 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:45.981314 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:45.981275 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:48.542388 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:48.542349 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:48.746322 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:48.746284 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:51.078109 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:51.078072 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:52.794112 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:52.794073 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:56.130422 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:19:56.130386 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:58.469566 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:19:58.469519 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:20:01.137694 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:01.137631 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:06.120544 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:06.120498 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:11.210863 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:11.210766 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:16.221057 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:16.221016 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:21.160061 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:21.160024 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:26.256664 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:26.256631 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:28.469549 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:20:28.469508 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:20:30.740153 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:30.740086 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:30.942491 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:30.942454 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:31.227110 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:31.227072 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:33.682760 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:33.682723 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:36.667869 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:36.667825 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:37.811387 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:37.811358 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:41.636887 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:41.636852 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:41.885399 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:41.885369 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:46.710540 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:46.710491 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:51.719325 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:51.719286 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:52.381173 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:52.381104 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d"] Nov 02 22:20:52.385612 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:52.385585 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" Nov 02 22:20:52.437552 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:52.437515 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d"] Nov 02 22:20:52.482079 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:52.482047 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-home\") pod \"test-model-dd28e6-kserve-7ffcf54cd9-2x66d\" (UID: \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\") " pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" Nov 02 22:20:52.482310 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:52.482120 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-model-cache\") pod \"test-model-dd28e6-kserve-7ffcf54cd9-2x66d\" (UID: \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\") " pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" Nov 02 22:20:52.482310 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:52.482262 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-dshm\") pod \"test-model-dd28e6-kserve-7ffcf54cd9-2x66d\" (UID: \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\") " pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" Nov 02 22:20:52.482412 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:52.482324 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-tmp-dir\") pod \"test-model-dd28e6-kserve-7ffcf54cd9-2x66d\" (UID: \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\") " pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" Nov 02 22:20:52.482412 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:52.482368 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-tls-certs\") pod \"test-model-dd28e6-kserve-7ffcf54cd9-2x66d\" (UID: \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\") " pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" Nov 02 22:20:52.482412 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:52.482394 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2vcw\" (UniqueName: \"kubernetes.io/projected/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-kube-api-access-g2vcw\") pod \"test-model-dd28e6-kserve-7ffcf54cd9-2x66d\" (UID: \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\") " pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" Nov 02 22:20:52.498508 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:52.498472 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-dd28e6\"/\"test-model-dd28e6-kserve-self-signed-certs\"" Nov 02 22:20:52.498663 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:52.498584 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-dd28e6\"/\"default-dockercfg-d8brc\"" Nov 02 22:20:52.513715 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:52.513689 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:52.514455 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:52.514437 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-dd28e6\"/\"openshift-service-ca.crt\"" Nov 02 22:20:52.514635 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:52.514620 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-dd28e6\"/\"kube-root-ca.crt\"" Nov 02 22:20:52.583081 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:52.583040 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-home\") pod \"test-model-dd28e6-kserve-7ffcf54cd9-2x66d\" (UID: \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\") " pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" Nov 02 22:20:52.583322 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:52.583165 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-model-cache\") pod \"test-model-dd28e6-kserve-7ffcf54cd9-2x66d\" (UID: \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\") " pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" Nov 02 22:20:52.583322 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:52.583208 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-dshm\") pod \"test-model-dd28e6-kserve-7ffcf54cd9-2x66d\" (UID: \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\") " pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" Nov 02 22:20:52.583322 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:52.583234 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-tmp-dir\") pod \"test-model-dd28e6-kserve-7ffcf54cd9-2x66d\" (UID: \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\") " pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" Nov 02 22:20:52.583322 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:52.583282 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-tls-certs\") pod \"test-model-dd28e6-kserve-7ffcf54cd9-2x66d\" (UID: \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\") " pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" Nov 02 22:20:52.583322 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:52.583319 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2vcw\" (UniqueName: \"kubernetes.io/projected/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-kube-api-access-g2vcw\") pod \"test-model-dd28e6-kserve-7ffcf54cd9-2x66d\" (UID: \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\") " pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" Nov 02 22:20:52.583601 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:52.583542 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-home\") pod \"test-model-dd28e6-kserve-7ffcf54cd9-2x66d\" (UID: \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\") " pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" Nov 02 22:20:52.583601 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:52.583592 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-model-cache\") pod \"test-model-dd28e6-kserve-7ffcf54cd9-2x66d\" (UID: \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\") " pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" Nov 02 22:20:52.584072 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:52.584048 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-tmp-dir\") pod \"test-model-dd28e6-kserve-7ffcf54cd9-2x66d\" (UID: \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\") " pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" Nov 02 22:20:52.586046 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:52.586022 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-tls-certs\") pod \"test-model-dd28e6-kserve-7ffcf54cd9-2x66d\" (UID: \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\") " pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" Nov 02 22:20:52.586255 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:52.586233 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-dshm\") pod \"test-model-dd28e6-kserve-7ffcf54cd9-2x66d\" (UID: \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\") " pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" Nov 02 22:20:52.636343 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:52.636267 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2vcw\" (UniqueName: \"kubernetes.io/projected/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-kube-api-access-g2vcw\") pod \"test-model-dd28e6-kserve-7ffcf54cd9-2x66d\" (UID: \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\") " pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" Nov 02 22:20:52.698298 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:52.698253 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" Nov 02 22:20:53.130656 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:53.130628 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d"] Nov 02 22:20:53.132228 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:20:53.132198 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b2d5ed1_0979_476f_bc8e_dfe5099b5f49.slice/crio-3b745fd87c59c9e3e5f78267db881935ee7e39ee13dcecb290a38b80406f351d WatchSource:0}: Error finding container 3b745fd87c59c9e3e5f78267db881935ee7e39ee13dcecb290a38b80406f351d: Status 404 returned error can't find the container with id 3b745fd87c59c9e3e5f78267db881935ee7e39ee13dcecb290a38b80406f351d Nov 02 22:20:53.798662 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:53.798626 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" event={"ID":"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49","Type":"ContainerStarted","Data":"2dd97b344c55c15cff538533d5ca45e7980ee8d6a1aa62044664389d7920e829"} Nov 02 22:20:53.798836 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:53.798677 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" event={"ID":"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49","Type":"ContainerStarted","Data":"3b745fd87c59c9e3e5f78267db881935ee7e39ee13dcecb290a38b80406f351d"} Nov 02 22:20:53.909006 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:53.908946 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" podStartSLOduration=1.9089281790000001 podStartE2EDuration="1.908928179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:53.904360924 +0000 UTC m=+4346.343935650" watchObservedRunningTime="2026-11-02 22:20:53.908928179 +0000 UTC m=+4346.348502902" Nov 02 22:20:54.032252 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:54.032211 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:54.235007 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:54.234907 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:54.699006 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:54.698971 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" Nov 02 22:20:55.883563 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:55.883528 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:56.289668 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:56.289609 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:56.940444 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:56.940376 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:57.150326 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:20:57.150282 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:58.473881 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:20:58.473839 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:21:00.957276 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:00.957245 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:01.177412 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:01.177375 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:02.314621 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:02.314576 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:02.698782 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:02.698705 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" Nov 02 22:21:02.711794 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:02.711767 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" Nov 02 22:21:02.873299 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:02.873263 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" Nov 02 22:21:02.962209 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:02.962087 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:03.569520 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:03.569463 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:03.902905 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:03.902788 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:04.128565 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:04.128523 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:04.334192 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:04.333993 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:07.650510 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:07.650470 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:07.900684 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:07.900602 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:08.123227 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:08.123156 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:10.792314 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:10.792268 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:10.994610 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:10.994561 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:11.200855 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:11.200771 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:13.058765 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:13.058724 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:18.039178 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:18.039106 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:23.127881 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:23.127844 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:28.005149 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:28.005064 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:28.468513 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:21:28.468469 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:21:32.977930 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:32.977891 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:35.245054 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:35.245014 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k"] Nov 02 22:21:35.250422 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:35.250397 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" Nov 02 22:21:35.300471 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:35.300436 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k"] Nov 02 22:21:35.334580 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:35.334537 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-home\") pod \"test-model-cf2a53-kserve-85658974dc-9zl8k\" (UID: \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\") " pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" Nov 02 22:21:35.334736 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:35.334707 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt6nc\" (UniqueName: \"kubernetes.io/projected/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-kube-api-access-bt6nc\") pod \"test-model-cf2a53-kserve-85658974dc-9zl8k\" (UID: \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\") " pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" Nov 02 22:21:35.334781 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:35.334757 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-tls-certs\") pod \"test-model-cf2a53-kserve-85658974dc-9zl8k\" (UID: \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\") " pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" Nov 02 22:21:35.334817 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:35.334802 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-dshm\") pod \"test-model-cf2a53-kserve-85658974dc-9zl8k\" (UID: \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\") " pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" Nov 02 22:21:35.334987 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:35.334971 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-tmp-dir\") pod \"test-model-cf2a53-kserve-85658974dc-9zl8k\" (UID: \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\") " pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" Nov 02 22:21:35.335048 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:35.335029 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-model-cache\") pod \"test-model-cf2a53-kserve-85658974dc-9zl8k\" (UID: \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\") " pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" Nov 02 22:21:35.342239 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:35.342203 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cf2a53\"/\"openshift-service-ca.crt\"" Nov 02 22:21:35.342358 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:35.342227 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cf2a53\"/\"kube-root-ca.crt\"" Nov 02 22:21:35.345357 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:35.345335 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cf2a53\"/\"default-dockercfg-4x8hr\"" Nov 02 22:21:35.345456 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:35.345362 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cf2a53\"/\"test-model-cf2a53-kserve-self-signed-certs\"" Nov 02 22:21:35.426109 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:35.426070 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:35.436549 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:35.436510 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-tmp-dir\") pod \"test-model-cf2a53-kserve-85658974dc-9zl8k\" (UID: \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\") " pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" Nov 02 22:21:35.436842 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:35.436815 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-model-cache\") pod \"test-model-cf2a53-kserve-85658974dc-9zl8k\" (UID: \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\") " pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" Nov 02 22:21:35.436979 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:35.436921 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-home\") pod \"test-model-cf2a53-kserve-85658974dc-9zl8k\" (UID: \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\") " pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" Nov 02 22:21:35.437050 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:35.436974 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-tmp-dir\") pod \"test-model-cf2a53-kserve-85658974dc-9zl8k\" (UID: \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\") " pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" Nov 02 22:21:35.437050 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:35.437016 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt6nc\" (UniqueName: \"kubernetes.io/projected/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-kube-api-access-bt6nc\") pod \"test-model-cf2a53-kserve-85658974dc-9zl8k\" (UID: \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\") " pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" Nov 02 22:21:35.437216 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:35.437060 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-tls-certs\") pod \"test-model-cf2a53-kserve-85658974dc-9zl8k\" (UID: \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\") " pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" Nov 02 22:21:35.437216 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:35.437085 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-dshm\") pod \"test-model-cf2a53-kserve-85658974dc-9zl8k\" (UID: \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\") " pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" Nov 02 22:21:35.437337 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:35.437220 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-model-cache\") pod \"test-model-cf2a53-kserve-85658974dc-9zl8k\" (UID: \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\") " pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" Nov 02 22:21:35.437337 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:35.437256 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-home\") pod \"test-model-cf2a53-kserve-85658974dc-9zl8k\" (UID: \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\") " pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" Nov 02 22:21:35.439487 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:35.439466 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-dshm\") pod \"test-model-cf2a53-kserve-85658974dc-9zl8k\" (UID: \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\") " pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" Nov 02 22:21:35.440067 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:35.440046 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-tls-certs\") pod \"test-model-cf2a53-kserve-85658974dc-9zl8k\" (UID: \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\") " pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" Nov 02 22:21:35.489801 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:35.489767 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt6nc\" (UniqueName: \"kubernetes.io/projected/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-kube-api-access-bt6nc\") pod \"test-model-cf2a53-kserve-85658974dc-9zl8k\" (UID: \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\") " pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" Nov 02 22:21:35.563917 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:35.563869 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" Nov 02 22:21:35.947962 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:35.947878 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k"] Nov 02 22:21:35.953933 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:21:35.953898 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda45b8c7b_0efd_414f_a246_0b8ecb6349e3.slice/crio-51baf54d8cdafd0147240612ff53367e97753f8a3c9856e16e7a57fd8a700072 WatchSource:0}: Error finding container 51baf54d8cdafd0147240612ff53367e97753f8a3c9856e16e7a57fd8a700072: Status 404 returned error can't find the container with id 51baf54d8cdafd0147240612ff53367e97753f8a3c9856e16e7a57fd8a700072 Nov 02 22:21:36.090903 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:36.090862 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" event={"ID":"a45b8c7b-0efd-414f-a246-0b8ecb6349e3","Type":"ContainerStarted","Data":"71d139f301bad4d04f48d87609f2ce27e37fa582cdffa2e683c7072d6bc9fd5a"} Nov 02 22:21:36.090903 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:36.090908 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" event={"ID":"a45b8c7b-0efd-414f-a246-0b8ecb6349e3","Type":"ContainerStarted","Data":"51baf54d8cdafd0147240612ff53367e97753f8a3c9856e16e7a57fd8a700072"} Nov 02 22:21:36.196818 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:36.196717 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" podStartSLOduration=1.196704534 podStartE2EDuration="1.196704534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:36.196414259 +0000 UTC m=+4388.635988983" watchObservedRunningTime="2026-11-02 22:21:36.196704534 +0000 UTC m=+4388.636279247" Nov 02 22:21:36.564391 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:36.564343 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" Nov 02 22:21:38.010781 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:38.010742 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:41.975581 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:41.975547 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:42.984900 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:42.984867 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp"] Nov 02 22:21:42.988821 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:42.988800 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" Nov 02 22:21:43.047203 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:43.047167 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp"] Nov 02 22:21:43.047993 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:43.047961 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:43.085677 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:43.085648 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-0f0874\"/\"openshift-service-ca.crt\"" Nov 02 22:21:43.085677 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:43.085673 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-0f0874\"/\"rate-test-model-0f0874-kserve-self-signed-certs\"" Nov 02 22:21:43.085937 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:43.085647 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-0f0874\"/\"kube-root-ca.crt\"" Nov 02 22:21:43.085937 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:43.085729 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-0f0874\"/\"default-dockercfg-r6mm5\"" Nov 02 22:21:43.126643 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:43.126616 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/244817d6-8757-4aab-bfed-038ee1a69bb9-dshm\") pod \"rate-test-model-0f0874-kserve-ff6c65965-qrqzp\" (UID: \"244817d6-8757-4aab-bfed-038ee1a69bb9\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" Nov 02 22:21:43.126793 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:43.126650 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/244817d6-8757-4aab-bfed-038ee1a69bb9-tls-certs\") pod \"rate-test-model-0f0874-kserve-ff6c65965-qrqzp\" (UID: \"244817d6-8757-4aab-bfed-038ee1a69bb9\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" Nov 02 22:21:43.126793 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:43.126685 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/244817d6-8757-4aab-bfed-038ee1a69bb9-model-cache\") pod \"rate-test-model-0f0874-kserve-ff6c65965-qrqzp\" (UID: \"244817d6-8757-4aab-bfed-038ee1a69bb9\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" Nov 02 22:21:43.126900 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:43.126876 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcbhk\" (UniqueName: \"kubernetes.io/projected/244817d6-8757-4aab-bfed-038ee1a69bb9-kube-api-access-fcbhk\") pod \"rate-test-model-0f0874-kserve-ff6c65965-qrqzp\" (UID: \"244817d6-8757-4aab-bfed-038ee1a69bb9\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" Nov 02 22:21:43.126947 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:43.126911 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/244817d6-8757-4aab-bfed-038ee1a69bb9-home\") pod \"rate-test-model-0f0874-kserve-ff6c65965-qrqzp\" (UID: \"244817d6-8757-4aab-bfed-038ee1a69bb9\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" Nov 02 22:21:43.127029 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:43.127010 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/244817d6-8757-4aab-bfed-038ee1a69bb9-tmp-dir\") pod \"rate-test-model-0f0874-kserve-ff6c65965-qrqzp\" (UID: \"244817d6-8757-4aab-bfed-038ee1a69bb9\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" Nov 02 22:21:43.228880 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:43.228823 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/244817d6-8757-4aab-bfed-038ee1a69bb9-tls-certs\") pod \"rate-test-model-0f0874-kserve-ff6c65965-qrqzp\" (UID: \"244817d6-8757-4aab-bfed-038ee1a69bb9\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" Nov 02 22:21:43.229047 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:43.228911 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/244817d6-8757-4aab-bfed-038ee1a69bb9-model-cache\") pod \"rate-test-model-0f0874-kserve-ff6c65965-qrqzp\" (UID: \"244817d6-8757-4aab-bfed-038ee1a69bb9\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" Nov 02 22:21:43.229144 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:43.229051 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcbhk\" (UniqueName: \"kubernetes.io/projected/244817d6-8757-4aab-bfed-038ee1a69bb9-kube-api-access-fcbhk\") pod \"rate-test-model-0f0874-kserve-ff6c65965-qrqzp\" (UID: \"244817d6-8757-4aab-bfed-038ee1a69bb9\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" Nov 02 22:21:43.229144 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:43.229085 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/244817d6-8757-4aab-bfed-038ee1a69bb9-home\") pod \"rate-test-model-0f0874-kserve-ff6c65965-qrqzp\" (UID: \"244817d6-8757-4aab-bfed-038ee1a69bb9\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" Nov 02 22:21:43.229261 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:43.229147 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/244817d6-8757-4aab-bfed-038ee1a69bb9-tmp-dir\") pod \"rate-test-model-0f0874-kserve-ff6c65965-qrqzp\" (UID: \"244817d6-8757-4aab-bfed-038ee1a69bb9\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" Nov 02 22:21:43.229261 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:43.229250 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/244817d6-8757-4aab-bfed-038ee1a69bb9-dshm\") pod \"rate-test-model-0f0874-kserve-ff6c65965-qrqzp\" (UID: \"244817d6-8757-4aab-bfed-038ee1a69bb9\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" Nov 02 22:21:43.230335 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:43.230305 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/244817d6-8757-4aab-bfed-038ee1a69bb9-tmp-dir\") pod \"rate-test-model-0f0874-kserve-ff6c65965-qrqzp\" (UID: \"244817d6-8757-4aab-bfed-038ee1a69bb9\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" Nov 02 22:21:43.230568 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:43.230549 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/244817d6-8757-4aab-bfed-038ee1a69bb9-model-cache\") pod \"rate-test-model-0f0874-kserve-ff6c65965-qrqzp\" (UID: \"244817d6-8757-4aab-bfed-038ee1a69bb9\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" Nov 02 22:21:43.230856 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:43.230839 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/244817d6-8757-4aab-bfed-038ee1a69bb9-home\") pod \"rate-test-model-0f0874-kserve-ff6c65965-qrqzp\" (UID: \"244817d6-8757-4aab-bfed-038ee1a69bb9\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" Nov 02 22:21:43.231629 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:43.231604 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/244817d6-8757-4aab-bfed-038ee1a69bb9-dshm\") pod \"rate-test-model-0f0874-kserve-ff6c65965-qrqzp\" (UID: \"244817d6-8757-4aab-bfed-038ee1a69bb9\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" Nov 02 22:21:43.232429 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:43.232407 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/244817d6-8757-4aab-bfed-038ee1a69bb9-tls-certs\") pod \"rate-test-model-0f0874-kserve-ff6c65965-qrqzp\" (UID: \"244817d6-8757-4aab-bfed-038ee1a69bb9\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" Nov 02 22:21:43.255763 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:43.255698 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:43.283731 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:43.283698 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcbhk\" (UniqueName: \"kubernetes.io/projected/244817d6-8757-4aab-bfed-038ee1a69bb9-kube-api-access-fcbhk\") pod \"rate-test-model-0f0874-kserve-ff6c65965-qrqzp\" (UID: \"244817d6-8757-4aab-bfed-038ee1a69bb9\") " pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" Nov 02 22:21:43.302530 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:43.302495 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" Nov 02 22:21:43.685965 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:43.685940 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp"] Nov 02 22:21:43.687663 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:21:43.687624 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod244817d6_8757_4aab_bfed_038ee1a69bb9.slice/crio-f510d045026652fc4064c5a63beda6761ca28fdcc50fffdf5e96f7f471db55bd WatchSource:0}: Error finding container f510d045026652fc4064c5a63beda6761ca28fdcc50fffdf5e96f7f471db55bd: Status 404 returned error can't find the container with id f510d045026652fc4064c5a63beda6761ca28fdcc50fffdf5e96f7f471db55bd Nov 02 22:21:44.156169 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:44.156111 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" event={"ID":"244817d6-8757-4aab-bfed-038ee1a69bb9","Type":"ContainerStarted","Data":"34f0f47b99a3cfef076073f2dc2b441e0e4431e534443270be55f3a9f97d4462"} Nov 02 22:21:44.156169 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:44.156168 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" event={"ID":"244817d6-8757-4aab-bfed-038ee1a69bb9","Type":"ContainerStarted","Data":"f510d045026652fc4064c5a63beda6761ca28fdcc50fffdf5e96f7f471db55bd"} Nov 02 22:21:44.265348 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:44.265295 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" podStartSLOduration=2.265281314 podStartE2EDuration="2.265281314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:44.261987582 +0000 UTC m=+4396.701562303" watchObservedRunningTime="2026-11-02 22:21:44.265281314 +0000 UTC m=+4396.704856036" Nov 02 22:21:44.303374 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:44.303345 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" Nov 02 22:21:45.564605 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:45.564567 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" Nov 02 22:21:45.577147 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:45.577100 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" Nov 02 22:21:46.180174 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:46.180124 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" Nov 02 22:21:48.020284 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:48.020250 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:53.016553 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:53.016514 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:53.303506 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:53.303420 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" Nov 02 22:21:53.315926 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:53.315901 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" Nov 02 22:21:54.235107 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:54.235062 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" Nov 02 22:21:58.088573 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:21:58.088539 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:58.469366 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:21:58.469282 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:22:03.124691 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:03.124645 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:06.922041 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:06.921997 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:08.090996 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:08.090947 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:13.100042 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:13.100000 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:18.145333 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:18.145290 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:23.214491 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:23.214441 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:28.079046 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:28.079005 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:28.286182 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:28.286145 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:28.471879 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:22:28.471786 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:22:28.491516 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:28.491480 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:33.230083 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:33.230025 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:37.822332 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:37.822295 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64"] Nov 02 22:22:37.826347 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:37.826323 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64" Nov 02 22:22:37.876725 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:37.876687 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b5e8794c-serving-cert\"" Nov 02 22:22:37.927970 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:37.927927 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64"] Nov 02 22:22:37.931188 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:37.931155 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93565534-5250-4709-8284-beb0aea24584-metrics-certs\") pod \"maas-api-e2e-mt-b5e8794c-8975d658f-9nn64\" (UID: \"93565534-5250-4709-8284-beb0aea24584\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64" Nov 02 22:22:37.931345 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:37.931290 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93565534-5250-4709-8284-beb0aea24584-maas-api-tls\") pod \"maas-api-e2e-mt-b5e8794c-8975d658f-9nn64\" (UID: \"93565534-5250-4709-8284-beb0aea24584\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64" Nov 02 22:22:37.931430 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:37.931409 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93565534-5250-4709-8284-beb0aea24584-openshift-service-ca\") pod \"maas-api-e2e-mt-b5e8794c-8975d658f-9nn64\" (UID: \"93565534-5250-4709-8284-beb0aea24584\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64" Nov 02 22:22:37.931578 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:37.931490 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlwpf\" (UniqueName: \"kubernetes.io/projected/93565534-5250-4709-8284-beb0aea24584-kube-api-access-jlwpf\") pod \"maas-api-e2e-mt-b5e8794c-8975d658f-9nn64\" (UID: \"93565534-5250-4709-8284-beb0aea24584\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64" Nov 02 22:22:38.033214 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:38.033108 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93565534-5250-4709-8284-beb0aea24584-openshift-service-ca\") pod \"maas-api-e2e-mt-b5e8794c-8975d658f-9nn64\" (UID: \"93565534-5250-4709-8284-beb0aea24584\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64" Nov 02 22:22:38.033214 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:38.033189 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlwpf\" (UniqueName: \"kubernetes.io/projected/93565534-5250-4709-8284-beb0aea24584-kube-api-access-jlwpf\") pod \"maas-api-e2e-mt-b5e8794c-8975d658f-9nn64\" (UID: \"93565534-5250-4709-8284-beb0aea24584\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64" Nov 02 22:22:38.033445 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:38.033327 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93565534-5250-4709-8284-beb0aea24584-metrics-certs\") pod \"maas-api-e2e-mt-b5e8794c-8975d658f-9nn64\" (UID: \"93565534-5250-4709-8284-beb0aea24584\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64" Nov 02 22:22:38.033445 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:38.033365 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93565534-5250-4709-8284-beb0aea24584-maas-api-tls\") pod \"maas-api-e2e-mt-b5e8794c-8975d658f-9nn64\" (UID: \"93565534-5250-4709-8284-beb0aea24584\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64" Nov 02 22:22:38.033564 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:22:38.033523 4935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-serving-cert: secret "maas-api-e2e-mt-b5e8794c-serving-cert" not found Nov 02 22:22:38.033669 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:22:38.033616 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93565534-5250-4709-8284-beb0aea24584-maas-api-tls podName:93565534-5250-4709-8284-beb0aea24584 nodeName:}" failed. No retries permitted until 2026-11-02 22:22:38.533595021 +0000 UTC m=+4450.973169734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/93565534-5250-4709-8284-beb0aea24584-maas-api-tls") pod "maas-api-e2e-mt-b5e8794c-8975d658f-9nn64" (UID: "93565534-5250-4709-8284-beb0aea24584") : secret "maas-api-e2e-mt-b5e8794c-serving-cert" not found Nov 02 22:22:38.033947 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:38.033924 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93565534-5250-4709-8284-beb0aea24584-openshift-service-ca\") pod \"maas-api-e2e-mt-b5e8794c-8975d658f-9nn64\" (UID: \"93565534-5250-4709-8284-beb0aea24584\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64" Nov 02 22:22:38.036074 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:38.036045 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93565534-5250-4709-8284-beb0aea24584-metrics-certs\") pod \"maas-api-e2e-mt-b5e8794c-8975d658f-9nn64\" (UID: \"93565534-5250-4709-8284-beb0aea24584\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64" Nov 02 22:22:38.087776 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:38.087693 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlwpf\" (UniqueName: \"kubernetes.io/projected/93565534-5250-4709-8284-beb0aea24584-kube-api-access-jlwpf\") pod \"maas-api-e2e-mt-b5e8794c-8975d658f-9nn64\" (UID: \"93565534-5250-4709-8284-beb0aea24584\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64" Nov 02 22:22:38.108165 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:38.108099 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:38.315931 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:38.315894 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:38.516337 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:38.516303 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:38.539630 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:38.539586 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93565534-5250-4709-8284-beb0aea24584-maas-api-tls\") pod \"maas-api-e2e-mt-b5e8794c-8975d658f-9nn64\" (UID: \"93565534-5250-4709-8284-beb0aea24584\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64" Nov 02 22:22:38.542304 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:38.542276 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93565534-5250-4709-8284-beb0aea24584-maas-api-tls\") pod \"maas-api-e2e-mt-b5e8794c-8975d658f-9nn64\" (UID: \"93565534-5250-4709-8284-beb0aea24584\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64" Nov 02 22:22:38.740277 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:38.740238 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64" Nov 02 22:22:39.110582 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:39.110547 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64"] Nov 02 22:22:39.112540 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:22:39.112508 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93565534_5250_4709_8284_beb0aea24584.slice/crio-39ab7cdfce0381f700e01026a5517d9b953d864accc028c24e3177f95134dfe6 WatchSource:0}: Error finding container 39ab7cdfce0381f700e01026a5517d9b953d864accc028c24e3177f95134dfe6: Status 404 returned error can't find the container with id 39ab7cdfce0381f700e01026a5517d9b953d864accc028c24e3177f95134dfe6 Nov 02 22:22:39.540553 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:39.540514 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64" event={"ID":"93565534-5250-4709-8284-beb0aea24584","Type":"ContainerStarted","Data":"39ab7cdfce0381f700e01026a5517d9b953d864accc028c24e3177f95134dfe6"} Nov 02 22:22:40.551216 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:40.551173 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64" event={"ID":"93565534-5250-4709-8284-beb0aea24584","Type":"ContainerStarted","Data":"054d2e5821daf8c4f8799d52f44890cc4a497c0e8e5b97c66183881c53feec16"} Nov 02 22:22:40.551745 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:40.551350 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64" Nov 02 22:22:40.660533 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:40.660469 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64" podStartSLOduration=2.789735301 podStartE2EDuration="3.660454047s" totalImagesPullingTime="870.718746ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:22:40.657753945 +0000 UTC m=+4453.097328668" watchObservedRunningTime="2026-11-02 22:22:40.660454047 +0000 UTC m=+4453.100028822" Nov 02 22:22:42.312280 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:42.312237 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:43.237013 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:43.236975 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:46.563211 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:46.563177 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64" Nov 02 22:22:48.240842 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:48.240803 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:49.006777 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:49.006743 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:53.243116 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:53.243075 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:58.227644 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:22:58.227583 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:58.472485 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:22:58.472447 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:23:03.318851 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:23:03.318801 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:08.332292 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:23:08.332245 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:13.307639 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:23:13.307602 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:28.470055 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:23:28.470011 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:23:28.597691 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:23:28.597662 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 22:23:28.607248 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:23:28.607225 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 22:23:28.632714 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:23:28.632693 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 22:23:28.641789 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:23:28.641771 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 22:23:35.085539 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:23:35.085497 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:38.351155 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:23:38.351092 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:42.674210 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:23:42.674163 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:43.340671 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:23:43.338445 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:48.383090 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:23:48.383044 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:53.328628 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:23:53.328592 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:58.316417 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:23:58.316376 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:58.470656 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:23:58.470612 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:24:03.294003 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:24:03.293965 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:04.620036 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:24:04.619995 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:08.330433 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:24:08.330316 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:10.529206 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:24:10.529166 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:12.034836 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:24:12.034798 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:12.366854 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:24:12.366747 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:12.586085 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:24:12.586045 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:13.882667 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:24:13.882622 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:16.182370 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:24:16.182332 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:17.890109 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:24:17.890067 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:18.105934 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:24:18.105876 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:19.354568 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:24:19.354523 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:23.722617 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:24:23.722566 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:24.353253 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:24:24.353195 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:27.097215 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:24:27.097183 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:27.323198 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:24:27.323112 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:28.469775 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:24:28.469736 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:24:29.339567 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:24:29.339534 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:34.324095 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:24:34.324031 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:39.432410 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:24:39.432365 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:42.997332 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:24:42.997285 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:44.318655 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:24:44.318623 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:49.261680 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:24:49.261631 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:54.208025 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:24:54.207987 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:58.476100 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:24:58.476052 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:24:59.229885 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:24:59.229849 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:04.222154 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:04.221973 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:09.220235 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:09.220180 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:14.204066 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:14.204019 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:19.228751 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:19.228714 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:22.763906 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:22.763860 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m"] Nov 02 22:25:22.764428 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:22.764114 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" podUID="2750bd81-031e-4b2c-bb07-a0ab009da8aa" containerName="istio-proxy" containerID="cri-o://be4bc50f94ecdc015b399593f4676d1d52f928eee472020cdd545e1e95bfd0d8" gracePeriod=30 Nov 02 22:25:22.808254 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:22.808217 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k"] Nov 02 22:25:22.808443 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:22.808419 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" podUID="232da504-01ae-4fb4-86cb-43c8ca0f6a7a" containerName="istio-proxy" containerID="cri-o://8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5" gracePeriod=30 Nov 02 22:25:23.619297 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:23.619249 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:25.438592 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:25.438554 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:26.261160 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:26.259310 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-f7c8k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:25:26.261160 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:26.259387 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" podUID="232da504-01ae-4fb4-86cb-43c8ca0f6a7a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:25:26.496935 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:26.496893 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-b9q8m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:25:26.497357 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:26.496966 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" podUID="2750bd81-031e-4b2c-bb07-a0ab009da8aa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:25:28.083902 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.083871 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:25:28.113348 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.113322 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:25:28.130116 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.130091 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-workload-socket\") pod \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " Nov 02 22:25:28.130270 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.130160 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istio-token\" (UniqueName: \"kubernetes.io/projected/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istio-token\") pod \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " Nov 02 22:25:28.130270 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.130192 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-workload-socket\") pod \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " Nov 02 22:25:28.130270 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.130248 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istio-data\") pod \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " Nov 02 22:25:28.130270 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.130268 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-workload-certs\") pod \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " Nov 02 22:25:28.130579 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.130301 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istio-token\" (UniqueName: \"kubernetes.io/projected/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istio-token\") pod \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " Nov 02 22:25:28.130579 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.130338 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2750bd81-031e-4b2c-bb07-a0ab009da8aa-kube-api-access-wn4jz\" (UniqueName: \"kubernetes.io/projected/2750bd81-031e-4b2c-bb07-a0ab009da8aa-kube-api-access-wn4jz\") pod \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " Nov 02 22:25:28.130579 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.130388 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-workload-socket" pod "2750bd81-031e-4b2c-bb07-a0ab009da8aa" (UID: "2750bd81-031e-4b2c-bb07-a0ab009da8aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.130579 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.130401 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-kube-api-access-78qbp\" (UniqueName: \"kubernetes.io/projected/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-kube-api-access-78qbp\") pod \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " Nov 02 22:25:28.130579 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.130473 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-credential-socket\") pod \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " Nov 02 22:25:28.130579 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.130559 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istio-data\") pod \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " Nov 02 22:25:28.130906 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.130594 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istio-data" pod "2750bd81-031e-4b2c-bb07-a0ab009da8aa" (UID: "2750bd81-031e-4b2c-bb07-a0ab009da8aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.130906 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.130622 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istiod-ca-cert\") pod \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " Nov 02 22:25:28.130906 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.130620 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-workload-certs" pod "2750bd81-031e-4b2c-bb07-a0ab009da8aa" (UID: "2750bd81-031e-4b2c-bb07-a0ab009da8aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.130906 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.130656 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istio-envoy\") pod \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " Nov 02 22:25:28.130906 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.130719 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-workload-certs\") pod \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " Nov 02 22:25:28.130906 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.130752 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istio-podinfo\") pod \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\" (UID: \"232da504-01ae-4fb4-86cb-43c8ca0f6a7a\") " Nov 02 22:25:28.130906 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.130777 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-credential-socket" pod "232da504-01ae-4fb4-86cb-43c8ca0f6a7a" (UID: "232da504-01ae-4fb4-86cb-43c8ca0f6a7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.130906 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.130784 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istiod-ca-cert\") pod \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " Nov 02 22:25:28.130906 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.130869 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-credential-socket\") pod \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " Nov 02 22:25:28.131431 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.130912 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istio-podinfo\") pod \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " Nov 02 22:25:28.131431 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.130947 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istio-envoy\") pod \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\" (UID: \"2750bd81-031e-4b2c-bb07-a0ab009da8aa\") " Nov 02 22:25:28.131431 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.130989 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istiod-ca-cert" pod "232da504-01ae-4fb4-86cb-43c8ca0f6a7a" (UID: "232da504-01ae-4fb4-86cb-43c8ca0f6a7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:28.131431 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.131069 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istiod-ca-cert" pod "2750bd81-031e-4b2c-bb07-a0ab009da8aa" (UID: "2750bd81-031e-4b2c-bb07-a0ab009da8aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:28.131650 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.131579 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.131650 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.131603 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.131650 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.131618 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.131650 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.131635 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.131880 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.131651 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.131880 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.131667 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.132302 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.132277 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istio-data" pod "232da504-01ae-4fb4-86cb-43c8ca0f6a7a" (UID: "232da504-01ae-4fb4-86cb-43c8ca0f6a7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.132778 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.132747 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-workload-certs" pod "232da504-01ae-4fb4-86cb-43c8ca0f6a7a" (UID: "232da504-01ae-4fb4-86cb-43c8ca0f6a7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.132871 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.132834 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-workload-socket" pod "232da504-01ae-4fb4-86cb-43c8ca0f6a7a" (UID: "232da504-01ae-4fb4-86cb-43c8ca0f6a7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.133012 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.132989 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-credential-socket" pod "2750bd81-031e-4b2c-bb07-a0ab009da8aa" (UID: "2750bd81-031e-4b2c-bb07-a0ab009da8aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.133315 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.133290 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-kube-api-access-78qbp" pod "232da504-01ae-4fb4-86cb-43c8ca0f6a7a" (UID: "232da504-01ae-4fb4-86cb-43c8ca0f6a7a"). InnerVolumeSpecName "kube-api-access-78qbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:28.133522 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.133499 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istio-token" pod "2750bd81-031e-4b2c-bb07-a0ab009da8aa" (UID: "2750bd81-031e-4b2c-bb07-a0ab009da8aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:28.133630 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.133533 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istio-token" pod "232da504-01ae-4fb4-86cb-43c8ca0f6a7a" (UID: "232da504-01ae-4fb4-86cb-43c8ca0f6a7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:28.133765 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.133745 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istio-envoy" pod "232da504-01ae-4fb4-86cb-43c8ca0f6a7a" (UID: "232da504-01ae-4fb4-86cb-43c8ca0f6a7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.133860 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.133844 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2750bd81-031e-4b2c-bb07-a0ab009da8aa-kube-api-access-wn4jz" pod "2750bd81-031e-4b2c-bb07-a0ab009da8aa" (UID: "2750bd81-031e-4b2c-bb07-a0ab009da8aa"). InnerVolumeSpecName "kube-api-access-wn4jz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:28.134886 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.134860 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istio-podinfo" pod "232da504-01ae-4fb4-86cb-43c8ca0f6a7a" (UID: "232da504-01ae-4fb4-86cb-43c8ca0f6a7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:25:28.134971 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.134947 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istio-envoy" pod "2750bd81-031e-4b2c-bb07-a0ab009da8aa" (UID: "2750bd81-031e-4b2c-bb07-a0ab009da8aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:28.135107 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.135086 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istio-podinfo" pod "2750bd81-031e-4b2c-bb07-a0ab009da8aa" (UID: "2750bd81-031e-4b2c-bb07-a0ab009da8aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:25:28.233468 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.233436 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.233468 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.233464 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.233468 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.233474 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.233692 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.233484 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.233692 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.233495 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.233692 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.233503 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.233692 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.233511 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.233692 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.233521 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.233692 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.233531 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2750bd81-031e-4b2c-bb07-a0ab009da8aa-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.233692 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.233541 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn4jz\" (UniqueName: \"kubernetes.io/projected/2750bd81-031e-4b2c-bb07-a0ab009da8aa-kube-api-access-wn4jz\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.233692 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.233549 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78qbp\" (UniqueName: \"kubernetes.io/projected/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-kube-api-access-78qbp\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.233692 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.233557 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232da504-01ae-4fb4-86cb-43c8ca0f6a7a-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.474118 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:25:28.474033 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:25:28.723808 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.723776 4935 generic.go:356] "Generic (PLEG): container finished" podID="2750bd81-031e-4b2c-bb07-a0ab009da8aa" containerID="be4bc50f94ecdc015b399593f4676d1d52f928eee472020cdd545e1e95bfd0d8" exitCode=0 Nov 02 22:25:28.723999 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.723822 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" Nov 02 22:25:28.723999 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.723857 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" event={"ID":"2750bd81-031e-4b2c-bb07-a0ab009da8aa","Type":"ContainerDied","Data":"be4bc50f94ecdc015b399593f4676d1d52f928eee472020cdd545e1e95bfd0d8"} Nov 02 22:25:28.723999 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.723910 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m" event={"ID":"2750bd81-031e-4b2c-bb07-a0ab009da8aa","Type":"ContainerDied","Data":"b6e7b88a65730057382fd1b34629b3cc5fcb69498d73aef2d387f48c0eb9d668"} Nov 02 22:25:28.723999 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.723938 4935 scope.go:122] "RemoveContainer" containerID="be4bc50f94ecdc015b399593f4676d1d52f928eee472020cdd545e1e95bfd0d8" Nov 02 22:25:28.725576 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.725552 4935 generic.go:356] "Generic (PLEG): container finished" podID="232da504-01ae-4fb4-86cb-43c8ca0f6a7a" containerID="8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5" exitCode=0 Nov 02 22:25:28.725744 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.725595 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" event={"ID":"232da504-01ae-4fb4-86cb-43c8ca0f6a7a","Type":"ContainerDied","Data":"8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5"} Nov 02 22:25:28.725744 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.725603 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" Nov 02 22:25:28.725744 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.725632 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k" event={"ID":"232da504-01ae-4fb4-86cb-43c8ca0f6a7a","Type":"ContainerDied","Data":"f46a817823e2e239ecc83108c8ac37ca84988deead3c63aff533550d06afc483"} Nov 02 22:25:28.734602 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.734582 4935 scope.go:122] "RemoveContainer" containerID="be4bc50f94ecdc015b399593f4676d1d52f928eee472020cdd545e1e95bfd0d8" Nov 02 22:25:28.734906 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:25:28.734879 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4bc50f94ecdc015b399593f4676d1d52f928eee472020cdd545e1e95bfd0d8\": container with ID starting with be4bc50f94ecdc015b399593f4676d1d52f928eee472020cdd545e1e95bfd0d8 not found: ID does not exist" containerID="be4bc50f94ecdc015b399593f4676d1d52f928eee472020cdd545e1e95bfd0d8" Nov 02 22:25:28.735018 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.734920 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4bc50f94ecdc015b399593f4676d1d52f928eee472020cdd545e1e95bfd0d8"} err="failed to get container status \"be4bc50f94ecdc015b399593f4676d1d52f928eee472020cdd545e1e95bfd0d8\": rpc error: code = NotFound desc = could not find container \"be4bc50f94ecdc015b399593f4676d1d52f928eee472020cdd545e1e95bfd0d8\": container with ID starting with be4bc50f94ecdc015b399593f4676d1d52f928eee472020cdd545e1e95bfd0d8 not found: ID does not exist" Nov 02 22:25:28.735018 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.734943 4935 scope.go:122] "RemoveContainer" containerID="8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5" Nov 02 22:25:28.745560 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.745543 4935 scope.go:122] "RemoveContainer" containerID="8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5" Nov 02 22:25:28.745804 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:25:28.745784 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5\": container with ID starting with 8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5 not found: ID does not exist" containerID="8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5" Nov 02 22:25:28.745852 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.745813 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5"} err="failed to get container status \"8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5\": rpc error: code = NotFound desc = could not find container \"8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5\": container with ID starting with 8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5 not found: ID does not exist" Nov 02 22:25:28.882359 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.882324 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k"] Nov 02 22:25:28.929984 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.929933 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-f7c8k"] Nov 02 22:25:28.990355 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.990266 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m"] Nov 02 22:25:28.993763 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:28.993733 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-b9q8m"] Nov 02 22:25:29.777468 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:29.777427 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="232da504-01ae-4fb4-86cb-43c8ca0f6a7a" path="/var/lib/kubelet/pods/232da504-01ae-4fb4-86cb-43c8ca0f6a7a/volumes" Nov 02 22:25:29.777974 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:29.777897 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2750bd81-031e-4b2c-bb07-a0ab009da8aa" path="/var/lib/kubelet/pods/2750bd81-031e-4b2c-bb07-a0ab009da8aa/volumes" Nov 02 22:25:30.423844 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:30.423806 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:30.649005 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:30.648968 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:31.992466 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:31.992430 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:32.469246 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:25:32.463630 4935 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232da504_01ae_4fb4_86cb_43c8ca0f6a7a.slice/crio-conmon-8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232da504_01ae_4fb4_86cb_43c8ca0f6a7a.slice/crio-8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:25:32.585278 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:32.585245 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:33.341961 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:33.341924 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:35.834698 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:35.834666 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:38.685587 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:25:38.685501 4935 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232da504_01ae_4fb4_86cb_43c8ca0f6a7a.slice/crio-8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232da504_01ae_4fb4_86cb_43c8ca0f6a7a.slice/crio-conmon-8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:25:39.993043 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:39.993010 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:40.204663 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:40.204629 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:40.414508 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:40.414467 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:40.985926 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:40.985887 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:42.481695 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:25:42.481653 4935 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232da504_01ae_4fb4_86cb_43c8ca0f6a7a.slice/crio-conmon-8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232da504_01ae_4fb4_86cb_43c8ca0f6a7a.slice/crio-8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:25:43.314014 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:43.313984 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:46.127937 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:46.127904 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:46.619569 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:46.619535 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:51.924015 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:51.923951 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:52.225332 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:52.225248 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:52.551859 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:25:52.551817 4935 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232da504_01ae_4fb4_86cb_43c8ca0f6a7a.slice/crio-8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232da504_01ae_4fb4_86cb_43c8ca0f6a7a.slice/crio-conmon-8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:25:53.681184 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:25:53.681141 4935 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232da504_01ae_4fb4_86cb_43c8ca0f6a7a.slice/crio-8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232da504_01ae_4fb4_86cb_43c8ca0f6a7a.slice/crio-conmon-8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:25:57.036696 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:25:57.036656 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:58.469838 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:25:58.469795 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:26:02.625990 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:02.625951 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:02.627405 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:26:02.627372 4935 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232da504_01ae_4fb4_86cb_43c8ca0f6a7a.slice/crio-8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232da504_01ae_4fb4_86cb_43c8ca0f6a7a.slice/crio-conmon-8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:26:05.960158 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:05.960108 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d"] Nov 02 22:26:06.055814 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:06.055774 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:07.656795 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:07.656759 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:08.736179 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:26:08.736139 4935 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232da504_01ae_4fb4_86cb_43c8ca0f6a7a.slice/crio-8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232da504_01ae_4fb4_86cb_43c8ca0f6a7a.slice/crio-conmon-8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:26:12.605768 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:12.605730 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:12.643402 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:26:12.643362 4935 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232da504_01ae_4fb4_86cb_43c8ca0f6a7a.slice/crio-conmon-8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232da504_01ae_4fb4_86cb_43c8ca0f6a7a.slice/crio-8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:26:17.567807 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:17.567767 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:20.984170 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:20.984109 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" podUID="4b2d5ed1-0979-476f-bc8e-dfe5099b5f49" containerName="main" containerID="cri-o://2dd97b344c55c15cff538533d5ca45e7980ee8d6a1aa62044664389d7920e829" gracePeriod=45 Nov 02 22:26:21.266883 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:21.266850 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" Nov 02 22:26:21.364441 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:21.364399 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-kube-api-access-g2vcw\" (UniqueName: \"kubernetes.io/projected/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-kube-api-access-g2vcw\") pod \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\" (UID: \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\") " Nov 02 22:26:21.364640 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:21.364451 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-tls-certs\" (UniqueName: \"kubernetes.io/secret/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-tls-certs\") pod \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\" (UID: \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\") " Nov 02 22:26:21.364640 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:21.364481 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-model-cache\") pod \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\" (UID: \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\") " Nov 02 22:26:21.364640 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:21.364577 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-home\" (UniqueName: \"kubernetes.io/empty-dir/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-home\") pod \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\" (UID: \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\") " Nov 02 22:26:21.364827 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:21.364687 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-dshm\") pod \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\" (UID: \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\") " Nov 02 22:26:21.364827 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:21.364740 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-tmp-dir\") pod \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\" (UID: \"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49\") " Nov 02 22:26:21.365000 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:21.364973 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-home" pod "4b2d5ed1-0979-476f-bc8e-dfe5099b5f49" (UID: "4b2d5ed1-0979-476f-bc8e-dfe5099b5f49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:21.365080 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:21.365056 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-model-cache" pod "4b2d5ed1-0979-476f-bc8e-dfe5099b5f49" (UID: "4b2d5ed1-0979-476f-bc8e-dfe5099b5f49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:21.365233 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:21.365210 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-tmp-dir" pod "4b2d5ed1-0979-476f-bc8e-dfe5099b5f49" (UID: "4b2d5ed1-0979-476f-bc8e-dfe5099b5f49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:21.365369 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:21.365352 4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-tmp-dir\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:21.365428 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:21.365375 4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-model-cache\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:21.365428 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:21.365391 4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-home\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:21.367359 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:21.367332 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-kube-api-access-g2vcw" pod "4b2d5ed1-0979-476f-bc8e-dfe5099b5f49" (UID: "4b2d5ed1-0979-476f-bc8e-dfe5099b5f49"). InnerVolumeSpecName "kube-api-access-g2vcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:21.367581 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:21.367561 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-dshm" pod "4b2d5ed1-0979-476f-bc8e-dfe5099b5f49" (UID: "4b2d5ed1-0979-476f-bc8e-dfe5099b5f49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:21.367684 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:21.367668 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-tls-certs" pod "4b2d5ed1-0979-476f-bc8e-dfe5099b5f49" (UID: "4b2d5ed1-0979-476f-bc8e-dfe5099b5f49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:21.466472 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:21.466435 4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-dshm\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:21.466472 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:21.466462 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2vcw\" (UniqueName: \"kubernetes.io/projected/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-kube-api-access-g2vcw\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:21.466472 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:21.466472 4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49-tls-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:22.105148 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:22.105053 4935 generic.go:356] "Generic (PLEG): container finished" podID="4b2d5ed1-0979-476f-bc8e-dfe5099b5f49" containerID="2dd97b344c55c15cff538533d5ca45e7980ee8d6a1aa62044664389d7920e829" exitCode=0 Nov 02 22:26:22.105148 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:22.105092 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" Nov 02 22:26:22.105672 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:22.105123 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" event={"ID":"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49","Type":"ContainerDied","Data":"2dd97b344c55c15cff538533d5ca45e7980ee8d6a1aa62044664389d7920e829"} Nov 02 22:26:22.105672 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:22.105185 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d" event={"ID":"4b2d5ed1-0979-476f-bc8e-dfe5099b5f49","Type":"ContainerDied","Data":"3b745fd87c59c9e3e5f78267db881935ee7e39ee13dcecb290a38b80406f351d"} Nov 02 22:26:22.105672 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:22.105208 4935 scope.go:122] "RemoveContainer" containerID="2dd97b344c55c15cff538533d5ca45e7980ee8d6a1aa62044664389d7920e829" Nov 02 22:26:22.115577 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:22.115555 4935 scope.go:122] "RemoveContainer" containerID="2dd97b344c55c15cff538533d5ca45e7980ee8d6a1aa62044664389d7920e829" Nov 02 22:26:22.115849 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:26:22.115830 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd97b344c55c15cff538533d5ca45e7980ee8d6a1aa62044664389d7920e829\": container with ID starting with 2dd97b344c55c15cff538533d5ca45e7980ee8d6a1aa62044664389d7920e829 not found: ID does not exist" containerID="2dd97b344c55c15cff538533d5ca45e7980ee8d6a1aa62044664389d7920e829" Nov 02 22:26:22.115896 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:22.115858 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd97b344c55c15cff538533d5ca45e7980ee8d6a1aa62044664389d7920e829"} err="failed to get container status \"2dd97b344c55c15cff538533d5ca45e7980ee8d6a1aa62044664389d7920e829\": rpc error: code = NotFound desc = could not find container \"2dd97b344c55c15cff538533d5ca45e7980ee8d6a1aa62044664389d7920e829\": container with ID starting with 2dd97b344c55c15cff538533d5ca45e7980ee8d6a1aa62044664389d7920e829 not found: ID does not exist" Nov 02 22:26:22.258934 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:22.258893 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d"] Nov 02 22:26:22.304975 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:22.304940 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-dd28e6/test-model-dd28e6-kserve-7ffcf54cd9-2x66d"] Nov 02 22:26:22.627975 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:22.627926 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:22.715635 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:26:22.715597 4935 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232da504_01ae_4fb4_86cb_43c8ca0f6a7a.slice/crio-8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232da504_01ae_4fb4_86cb_43c8ca0f6a7a.slice/crio-conmon-8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:26:22.827914 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:22.827877 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:22.985314 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:26:22.985216 4935 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232da504_01ae_4fb4_86cb_43c8ca0f6a7a.slice/crio-conmon-8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232da504_01ae_4fb4_86cb_43c8ca0f6a7a.slice/crio-8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:26:22.985865 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:26:22.985833 4935 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232da504_01ae_4fb4_86cb_43c8ca0f6a7a.slice/crio-8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232da504_01ae_4fb4_86cb_43c8ca0f6a7a.slice/crio-conmon-8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:26:23.676889 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:26:23.676847 4935 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232da504_01ae_4fb4_86cb_43c8ca0f6a7a.slice/crio-8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232da504_01ae_4fb4_86cb_43c8ca0f6a7a.slice/crio-conmon-8038678e7ee0ddd6fc999bd77a47326149c064c0a303b78508a5fef0088168d5.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:26:23.777363 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:23.777327 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b2d5ed1-0979-476f-bc8e-dfe5099b5f49" path="/var/lib/kubelet/pods/4b2d5ed1-0979-476f-bc8e-dfe5099b5f49/volumes" Nov 02 22:26:27.609943 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:27.609904 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:28.469706 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:26:28.469662 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:26:32.608249 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:32.608212 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:37.483046 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:37.483002 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:37.708986 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:37.708916 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:41.051738 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:41.051694 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:42.632046 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:42.631986 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:43.673482 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:43.673441 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:46.266440 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:46.266385 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k"] Nov 02 22:26:46.379154 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:46.376526 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:46.809312 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:46.809273 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:47.623657 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:47.623627 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:52.635098 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:52.635052 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:52.861960 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:52.861928 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:53.740611 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:53.740572 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:53.962532 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:53.962496 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:57.848228 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:57.848176 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:58.078737 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:58.078701 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:58.468480 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:26:58.468445 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:26:58.488063 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:26:58.488035 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:01.287438 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:01.287371 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" podUID="a45b8c7b-0efd-414f-a246-0b8ecb6349e3" containerName="main" containerID="cri-o://71d139f301bad4d04f48d87609f2ce27e37fa582cdffa2e683c7072d6bc9fd5a" gracePeriod=45 Nov 02 22:27:01.568787 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:01.568755 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" Nov 02 22:27:01.627051 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:01.627003 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-dshm\") pod \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\" (UID: \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\") " Nov 02 22:27:01.627266 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:01.627071 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-tls-certs\" (UniqueName: \"kubernetes.io/secret/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-tls-certs\") pod \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\" (UID: \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\") " Nov 02 22:27:01.627266 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:01.627142 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-home\" (UniqueName: \"kubernetes.io/empty-dir/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-home\") pod \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\" (UID: \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\") " Nov 02 22:27:01.627266 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:01.627220 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-model-cache\") pod \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\" (UID: \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\") " Nov 02 22:27:01.627444 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:01.627298 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-kube-api-access-bt6nc\" (UniqueName: \"kubernetes.io/projected/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-kube-api-access-bt6nc\") pod \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\" (UID: \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\") " Nov 02 22:27:01.627444 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:01.627328 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-tmp-dir\") pod \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\" (UID: \"a45b8c7b-0efd-414f-a246-0b8ecb6349e3\") " Nov 02 22:27:01.627610 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:01.627426 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-home" pod "a45b8c7b-0efd-414f-a246-0b8ecb6349e3" (UID: "a45b8c7b-0efd-414f-a246-0b8ecb6349e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:01.627610 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:01.627475 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-model-cache" pod "a45b8c7b-0efd-414f-a246-0b8ecb6349e3" (UID: "a45b8c7b-0efd-414f-a246-0b8ecb6349e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:01.627838 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:01.627802 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-tmp-dir" pod "a45b8c7b-0efd-414f-a246-0b8ecb6349e3" (UID: "a45b8c7b-0efd-414f-a246-0b8ecb6349e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:01.627838 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:01.627821 4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-home\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:01.628053 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:01.627859 4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-model-cache\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:01.629529 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:01.629504 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-tls-certs" pod "a45b8c7b-0efd-414f-a246-0b8ecb6349e3" (UID: "a45b8c7b-0efd-414f-a246-0b8ecb6349e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:27:01.629529 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:01.629513 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-dshm" pod "a45b8c7b-0efd-414f-a246-0b8ecb6349e3" (UID: "a45b8c7b-0efd-414f-a246-0b8ecb6349e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:01.629859 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:01.629832 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-kube-api-access-bt6nc" pod "a45b8c7b-0efd-414f-a246-0b8ecb6349e3" (UID: "a45b8c7b-0efd-414f-a246-0b8ecb6349e3"). InnerVolumeSpecName "kube-api-access-bt6nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:01.729267 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:01.729225 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt6nc\" (UniqueName: \"kubernetes.io/projected/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-kube-api-access-bt6nc\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:01.729267 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:01.729258 4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-tmp-dir\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:01.729267 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:01.729271 4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-dshm\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:01.729525 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:01.729284 4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a45b8c7b-0efd-414f-a246-0b8ecb6349e3-tls-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:02.300532 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:02.300496 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:02.380548 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:02.380513 4935 generic.go:356] "Generic (PLEG): container finished" podID="a45b8c7b-0efd-414f-a246-0b8ecb6349e3" containerID="71d139f301bad4d04f48d87609f2ce27e37fa582cdffa2e683c7072d6bc9fd5a" exitCode=0 Nov 02 22:27:02.380768 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:02.380566 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" Nov 02 22:27:02.380768 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:02.380601 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" event={"ID":"a45b8c7b-0efd-414f-a246-0b8ecb6349e3","Type":"ContainerDied","Data":"71d139f301bad4d04f48d87609f2ce27e37fa582cdffa2e683c7072d6bc9fd5a"} Nov 02 22:27:02.380768 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:02.380654 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k" event={"ID":"a45b8c7b-0efd-414f-a246-0b8ecb6349e3","Type":"ContainerDied","Data":"51baf54d8cdafd0147240612ff53367e97753f8a3c9856e16e7a57fd8a700072"} Nov 02 22:27:02.380768 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:02.380677 4935 scope.go:122] "RemoveContainer" containerID="71d139f301bad4d04f48d87609f2ce27e37fa582cdffa2e683c7072d6bc9fd5a" Nov 02 22:27:02.393218 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:02.393176 4935 scope.go:122] "RemoveContainer" containerID="71d139f301bad4d04f48d87609f2ce27e37fa582cdffa2e683c7072d6bc9fd5a" Nov 02 22:27:02.393579 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:27:02.393550 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71d139f301bad4d04f48d87609f2ce27e37fa582cdffa2e683c7072d6bc9fd5a\": container with ID starting with 71d139f301bad4d04f48d87609f2ce27e37fa582cdffa2e683c7072d6bc9fd5a not found: ID does not exist" containerID="71d139f301bad4d04f48d87609f2ce27e37fa582cdffa2e683c7072d6bc9fd5a" Nov 02 22:27:02.393656 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:02.393593 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d139f301bad4d04f48d87609f2ce27e37fa582cdffa2e683c7072d6bc9fd5a"} err="failed to get container status \"71d139f301bad4d04f48d87609f2ce27e37fa582cdffa2e683c7072d6bc9fd5a\": rpc error: code = NotFound desc = could not find container \"71d139f301bad4d04f48d87609f2ce27e37fa582cdffa2e683c7072d6bc9fd5a\": container with ID starting with 71d139f301bad4d04f48d87609f2ce27e37fa582cdffa2e683c7072d6bc9fd5a not found: ID does not exist" Nov 02 22:27:02.536844 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:02.536802 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k"] Nov 02 22:27:02.583509 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:02.583426 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-cf2a53/test-model-cf2a53-kserve-85658974dc-9zl8k"] Nov 02 22:27:03.777801 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:03.777760 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a45b8c7b-0efd-414f-a246-0b8ecb6349e3" path="/var/lib/kubelet/pods/a45b8c7b-0efd-414f-a246-0b8ecb6349e3/volumes" Nov 02 22:27:04.230235 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:04.230107 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:04.947086 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:04.947055 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:05.531022 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:05.530980 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:06.440599 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:06.440566 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:06.664097 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:06.664061 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:06.890197 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:06.890164 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:08.955911 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:08.955829 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64"] Nov 02 22:27:08.956351 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:08.956056 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64" podUID="93565534-5250-4709-8284-beb0aea24584" containerName="maas-api" containerID="cri-o://054d2e5821daf8c4f8799d52f44890cc4a497c0e8e5b97c66183881c53feec16" gracePeriod=30 Nov 02 22:27:09.159167 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:09.158811 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:09.239487 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:09.239460 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64" Nov 02 22:27:09.317159 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:09.317093 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93565534-5250-4709-8284-beb0aea24584-kube-api-access-jlwpf\" (UniqueName: \"kubernetes.io/projected/93565534-5250-4709-8284-beb0aea24584-kube-api-access-jlwpf\") pod \"93565534-5250-4709-8284-beb0aea24584\" (UID: \"93565534-5250-4709-8284-beb0aea24584\") " Nov 02 22:27:09.317358 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:09.317236 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93565534-5250-4709-8284-beb0aea24584-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93565534-5250-4709-8284-beb0aea24584-openshift-service-ca\") pod \"93565534-5250-4709-8284-beb0aea24584\" (UID: \"93565534-5250-4709-8284-beb0aea24584\") " Nov 02 22:27:09.317436 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:09.317356 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93565534-5250-4709-8284-beb0aea24584-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93565534-5250-4709-8284-beb0aea24584-maas-api-tls\") pod \"93565534-5250-4709-8284-beb0aea24584\" (UID: \"93565534-5250-4709-8284-beb0aea24584\") " Nov 02 22:27:09.317528 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:09.317506 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93565534-5250-4709-8284-beb0aea24584-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93565534-5250-4709-8284-beb0aea24584-metrics-certs\") pod \"93565534-5250-4709-8284-beb0aea24584\" (UID: \"93565534-5250-4709-8284-beb0aea24584\") " Nov 02 22:27:09.317676 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:09.317585 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93565534-5250-4709-8284-beb0aea24584-openshift-service-ca" pod "93565534-5250-4709-8284-beb0aea24584" (UID: "93565534-5250-4709-8284-beb0aea24584"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:09.317912 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:09.317880 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93565534-5250-4709-8284-beb0aea24584-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:09.319533 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:09.319505 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93565534-5250-4709-8284-beb0aea24584-kube-api-access-jlwpf" pod "93565534-5250-4709-8284-beb0aea24584" (UID: "93565534-5250-4709-8284-beb0aea24584"). InnerVolumeSpecName "kube-api-access-jlwpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:09.319654 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:09.319620 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93565534-5250-4709-8284-beb0aea24584-metrics-certs" pod "93565534-5250-4709-8284-beb0aea24584" (UID: "93565534-5250-4709-8284-beb0aea24584"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:27:09.319897 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:09.319869 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93565534-5250-4709-8284-beb0aea24584-maas-api-tls" pod "93565534-5250-4709-8284-beb0aea24584" (UID: "93565534-5250-4709-8284-beb0aea24584"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:27:09.419261 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:09.419227 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93565534-5250-4709-8284-beb0aea24584-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:09.419261 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:09.419255 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlwpf\" (UniqueName: \"kubernetes.io/projected/93565534-5250-4709-8284-beb0aea24584-kube-api-access-jlwpf\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:09.419261 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:09.419266 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93565534-5250-4709-8284-beb0aea24584-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:09.431896 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:09.431863 4935 generic.go:356] "Generic (PLEG): container finished" podID="93565534-5250-4709-8284-beb0aea24584" containerID="054d2e5821daf8c4f8799d52f44890cc4a497c0e8e5b97c66183881c53feec16" exitCode=0 Nov 02 22:27:09.432075 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:09.431920 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64" Nov 02 22:27:09.432075 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:09.431948 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64" event={"ID":"93565534-5250-4709-8284-beb0aea24584","Type":"ContainerDied","Data":"054d2e5821daf8c4f8799d52f44890cc4a497c0e8e5b97c66183881c53feec16"} Nov 02 22:27:09.432075 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:09.432004 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64" event={"ID":"93565534-5250-4709-8284-beb0aea24584","Type":"ContainerDied","Data":"39ab7cdfce0381f700e01026a5517d9b953d864accc028c24e3177f95134dfe6"} Nov 02 22:27:09.432075 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:09.432029 4935 scope.go:122] "RemoveContainer" containerID="054d2e5821daf8c4f8799d52f44890cc4a497c0e8e5b97c66183881c53feec16" Nov 02 22:27:09.443189 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:09.443164 4935 scope.go:122] "RemoveContainer" containerID="054d2e5821daf8c4f8799d52f44890cc4a497c0e8e5b97c66183881c53feec16" Nov 02 22:27:09.443451 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:27:09.443432 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"054d2e5821daf8c4f8799d52f44890cc4a497c0e8e5b97c66183881c53feec16\": container with ID starting with 054d2e5821daf8c4f8799d52f44890cc4a497c0e8e5b97c66183881c53feec16 not found: ID does not exist" containerID="054d2e5821daf8c4f8799d52f44890cc4a497c0e8e5b97c66183881c53feec16" Nov 02 22:27:09.443508 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:09.443459 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"054d2e5821daf8c4f8799d52f44890cc4a497c0e8e5b97c66183881c53feec16"} err="failed to get container status \"054d2e5821daf8c4f8799d52f44890cc4a497c0e8e5b97c66183881c53feec16\": rpc error: code = NotFound desc = could not find container \"054d2e5821daf8c4f8799d52f44890cc4a497c0e8e5b97c66183881c53feec16\": container with ID starting with 054d2e5821daf8c4f8799d52f44890cc4a497c0e8e5b97c66183881c53feec16 not found: ID does not exist" Nov 02 22:27:09.587028 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:09.586992 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64"] Nov 02 22:27:09.591570 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:09.591536 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b5e8794c-8975d658f-9nn64"] Nov 02 22:27:09.751867 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:09.751832 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:09.777303 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:09.777273 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93565534-5250-4709-8284-beb0aea24584" path="/var/lib/kubelet/pods/93565534-5250-4709-8284-beb0aea24584/volumes" Nov 02 22:27:09.995207 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:09.986352 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:10.991846 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:10.991813 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:11.236375 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:11.236337 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:11.487823 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:11.487761 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:12.725175 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:12.725122 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:12.950625 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:12.950581 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:13.241114 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:13.241080 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:14.383151 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:14.383104 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:14.612273 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:14.612232 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:16.535497 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:16.535443 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:18.941886 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:18.941850 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:19.867365 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:19.867264 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:20.082103 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:20.082067 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:21.886701 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:21.886660 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:24.603971 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:24.603928 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:26.894702 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:26.894669 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:27.129034 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:27.128973 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:28.469636 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:27:28.469591 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:27:31.955019 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:31.954978 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:36.324969 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:36.324920 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:38.215950 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:38.215898 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:38.819145 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:38.819094 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:39.120290 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:39.120197 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:40.482842 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:40.482809 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:40.968552 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:40.968510 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:41.687472 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:41.687430 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:42.648333 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:42.648296 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7"] Nov 02 22:27:42.648615 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:42.648583 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7" podUID="fa69559b-d902-4486-92e3-32af344bf33f" containerName="maas-api" containerID="cri-o://68edf1ffcc942c2e08ff11cf95c654c336640bfb0aa1efb9bd4db1681241ac6d" gracePeriod=30 Nov 02 22:27:42.840034 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:42.839958 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:42.929743 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:42.929711 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7" Nov 02 22:27:43.034114 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:43.034070 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa69559b-d902-4486-92e3-32af344bf33f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa69559b-d902-4486-92e3-32af344bf33f-maas-api-tls\") pod \"fa69559b-d902-4486-92e3-32af344bf33f\" (UID: \"fa69559b-d902-4486-92e3-32af344bf33f\") " Nov 02 22:27:43.034346 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:43.034212 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa69559b-d902-4486-92e3-32af344bf33f-kube-api-access-nxbdb\" (UniqueName: \"kubernetes.io/projected/fa69559b-d902-4486-92e3-32af344bf33f-kube-api-access-nxbdb\") pod \"fa69559b-d902-4486-92e3-32af344bf33f\" (UID: \"fa69559b-d902-4486-92e3-32af344bf33f\") " Nov 02 22:27:43.034346 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:43.034292 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa69559b-d902-4486-92e3-32af344bf33f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa69559b-d902-4486-92e3-32af344bf33f-openshift-service-ca\") pod \"fa69559b-d902-4486-92e3-32af344bf33f\" (UID: \"fa69559b-d902-4486-92e3-32af344bf33f\") " Nov 02 22:27:43.034487 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:43.034374 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa69559b-d902-4486-92e3-32af344bf33f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa69559b-d902-4486-92e3-32af344bf33f-metrics-certs\") pod \"fa69559b-d902-4486-92e3-32af344bf33f\" (UID: \"fa69559b-d902-4486-92e3-32af344bf33f\") " Nov 02 22:27:43.035029 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:43.034996 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa69559b-d902-4486-92e3-32af344bf33f-openshift-service-ca" pod "fa69559b-d902-4486-92e3-32af344bf33f" (UID: "fa69559b-d902-4486-92e3-32af344bf33f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:43.056202 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:43.038445 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa69559b-d902-4486-92e3-32af344bf33f-kube-api-access-nxbdb" pod "fa69559b-d902-4486-92e3-32af344bf33f" (UID: "fa69559b-d902-4486-92e3-32af344bf33f"). InnerVolumeSpecName "kube-api-access-nxbdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:43.056202 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:43.038546 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa69559b-d902-4486-92e3-32af344bf33f-metrics-certs" pod "fa69559b-d902-4486-92e3-32af344bf33f" (UID: "fa69559b-d902-4486-92e3-32af344bf33f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:27:43.071177 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:43.061346 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa69559b-d902-4486-92e3-32af344bf33f-maas-api-tls" pod "fa69559b-d902-4486-92e3-32af344bf33f" (UID: "fa69559b-d902-4486-92e3-32af344bf33f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:27:43.136429 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:43.136306 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa69559b-d902-4486-92e3-32af344bf33f-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:43.136429 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:43.136353 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa69559b-d902-4486-92e3-32af344bf33f-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:43.136429 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:43.136370 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxbdb\" (UniqueName: \"kubernetes.io/projected/fa69559b-d902-4486-92e3-32af344bf33f-kube-api-access-nxbdb\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:43.136429 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:43.136385 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa69559b-d902-4486-92e3-32af344bf33f-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:43.667716 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:43.667674 4935 generic.go:356] "Generic (PLEG): container finished" podID="fa69559b-d902-4486-92e3-32af344bf33f" containerID="68edf1ffcc942c2e08ff11cf95c654c336640bfb0aa1efb9bd4db1681241ac6d" exitCode=0 Nov 02 22:27:43.668029 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:43.668001 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7" event={"ID":"fa69559b-d902-4486-92e3-32af344bf33f","Type":"ContainerDied","Data":"68edf1ffcc942c2e08ff11cf95c654c336640bfb0aa1efb9bd4db1681241ac6d"} Nov 02 22:27:43.672340 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:43.672305 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7" event={"ID":"fa69559b-d902-4486-92e3-32af344bf33f","Type":"ContainerDied","Data":"fc1a4686d328ace91b696a661c6352d169399609e1b87b53604fda74d4ba599d"} Nov 02 22:27:43.672481 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:43.672361 4935 scope.go:122] "RemoveContainer" containerID="68edf1ffcc942c2e08ff11cf95c654c336640bfb0aa1efb9bd4db1681241ac6d" Nov 02 22:27:43.672481 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:43.668067 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7" Nov 02 22:27:43.686878 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:43.686854 4935 scope.go:122] "RemoveContainer" containerID="68edf1ffcc942c2e08ff11cf95c654c336640bfb0aa1efb9bd4db1681241ac6d" Nov 02 22:27:43.687234 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:27:43.687211 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68edf1ffcc942c2e08ff11cf95c654c336640bfb0aa1efb9bd4db1681241ac6d\": container with ID starting with 68edf1ffcc942c2e08ff11cf95c654c336640bfb0aa1efb9bd4db1681241ac6d not found: ID does not exist" containerID="68edf1ffcc942c2e08ff11cf95c654c336640bfb0aa1efb9bd4db1681241ac6d" Nov 02 22:27:43.687326 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:43.687246 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68edf1ffcc942c2e08ff11cf95c654c336640bfb0aa1efb9bd4db1681241ac6d"} err="failed to get container status \"68edf1ffcc942c2e08ff11cf95c654c336640bfb0aa1efb9bd4db1681241ac6d\": rpc error: code = NotFound desc = could not find container \"68edf1ffcc942c2e08ff11cf95c654c336640bfb0aa1efb9bd4db1681241ac6d\": container with ID starting with 68edf1ffcc942c2e08ff11cf95c654c336640bfb0aa1efb9bd4db1681241ac6d not found: ID does not exist" Nov 02 22:27:43.827267 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:43.827230 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7"] Nov 02 22:27:43.873416 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:43.873381 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dd28e6-5d75859fdc-dkcs7"] Nov 02 22:27:44.010885 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:44.010796 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:44.771064 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:44.771025 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:45.776655 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:45.776619 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa69559b-d902-4486-92e3-32af344bf33f" path="/var/lib/kubelet/pods/fa69559b-d902-4486-92e3-32af344bf33f/volumes" Nov 02 22:27:47.373480 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:47.373443 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:48.155898 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:48.155863 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:48.392844 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:48.392805 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:48.622323 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:48.622291 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:48.842342 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:48.842307 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:49.657623 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:49.657587 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:49.944962 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:49.944872 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:50.176326 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:50.176279 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:50.894023 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:50.893989 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:52.962119 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:52.962078 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:55.672753 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:55.672704 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:56.661465 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:56.661431 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:58.469690 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:27:58.469652 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:27:59.330322 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:59.330290 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:59.552607 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:27:59.552573 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:00.968043 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:00.968010 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:02.172303 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:02.172269 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:05.937577 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:05.937540 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:08.518236 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:08.518194 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:08.737640 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:08.737592 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:10.878066 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:10.878028 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:15.876805 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:15.876764 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:20.831115 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:20.831079 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:25.100608 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:25.100549 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:25.515145 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:25.515093 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:25.827544 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:25.827455 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:28.468944 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:28:28.468904 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:28:28.640839 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:28.640813 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 22:28:28.649960 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:28.649935 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 22:28:28.679481 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:28.679453 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 22:28:28.688511 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:28.688489 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 22:28:30.821380 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:30.821342 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:31.472734 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:31.472693 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:32.361339 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:32.361296 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:32.571962 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:32.571923 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:33.228606 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:33.228570 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:33.470791 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:33.470748 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:36.154867 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:36.154833 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:37.960159 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:37.960051 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:41.626542 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:41.626507 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:44.332914 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:44.332869 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:47.222934 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:47.222883 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:52.474793 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:52.474753 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:56.815672 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:56.815636 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:57.793670 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:57.793631 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt"] Nov 02 22:28:57.793860 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:57.793801 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt" podUID="7e3aac8c-f5ba-47d1-aae1-5baf948384b5" containerName="maas-api" containerID="cri-o://2cf98ebbadfc391dc269818c7470cdea134ef92a44869d28d5814c5a5f7c9b23" gracePeriod=30 Nov 02 22:28:58.040196 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:58.040112 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:58.087908 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:58.087863 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt" Nov 02 22:28:58.145959 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:58.145920 4935 generic.go:356] "Generic (PLEG): container finished" podID="7e3aac8c-f5ba-47d1-aae1-5baf948384b5" containerID="2cf98ebbadfc391dc269818c7470cdea134ef92a44869d28d5814c5a5f7c9b23" exitCode=0 Nov 02 22:28:58.145959 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:58.145965 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt" Nov 02 22:28:58.146320 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:58.146067 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt" event={"ID":"7e3aac8c-f5ba-47d1-aae1-5baf948384b5","Type":"ContainerDied","Data":"2cf98ebbadfc391dc269818c7470cdea134ef92a44869d28d5814c5a5f7c9b23"} Nov 02 22:28:58.146320 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:58.146116 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt" event={"ID":"7e3aac8c-f5ba-47d1-aae1-5baf948384b5","Type":"ContainerDied","Data":"f70d196da05aa23139a0573e02afd88a2f1846846cbe0684c8d1c48ab2ecb007"} Nov 02 22:28:58.146320 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:58.146162 4935 scope.go:122] "RemoveContainer" containerID="2cf98ebbadfc391dc269818c7470cdea134ef92a44869d28d5814c5a5f7c9b23" Nov 02 22:28:58.159040 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:58.159016 4935 scope.go:122] "RemoveContainer" containerID="2cf98ebbadfc391dc269818c7470cdea134ef92a44869d28d5814c5a5f7c9b23" Nov 02 22:28:58.159410 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:28:58.159383 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cf98ebbadfc391dc269818c7470cdea134ef92a44869d28d5814c5a5f7c9b23\": container with ID starting with 2cf98ebbadfc391dc269818c7470cdea134ef92a44869d28d5814c5a5f7c9b23 not found: ID does not exist" containerID="2cf98ebbadfc391dc269818c7470cdea134ef92a44869d28d5814c5a5f7c9b23" Nov 02 22:28:58.159506 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:58.159421 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cf98ebbadfc391dc269818c7470cdea134ef92a44869d28d5814c5a5f7c9b23"} err="failed to get container status \"2cf98ebbadfc391dc269818c7470cdea134ef92a44869d28d5814c5a5f7c9b23\": rpc error: code = NotFound desc = could not find container \"2cf98ebbadfc391dc269818c7470cdea134ef92a44869d28d5814c5a5f7c9b23\": container with ID starting with 2cf98ebbadfc391dc269818c7470cdea134ef92a44869d28d5814c5a5f7c9b23 not found: ID does not exist" Nov 02 22:28:58.209368 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:58.209330 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-openshift-service-ca\") pod \"7e3aac8c-f5ba-47d1-aae1-5baf948384b5\" (UID: \"7e3aac8c-f5ba-47d1-aae1-5baf948384b5\") " Nov 02 22:28:58.209550 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:58.209402 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-kube-api-access-cwsdv\" (UniqueName: \"kubernetes.io/projected/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-kube-api-access-cwsdv\") pod \"7e3aac8c-f5ba-47d1-aae1-5baf948384b5\" (UID: \"7e3aac8c-f5ba-47d1-aae1-5baf948384b5\") " Nov 02 22:28:58.209626 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:58.209557 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-maas-api-tls\") pod \"7e3aac8c-f5ba-47d1-aae1-5baf948384b5\" (UID: \"7e3aac8c-f5ba-47d1-aae1-5baf948384b5\") " Nov 02 22:28:58.209626 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:58.209616 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-metrics-certs\") pod \"7e3aac8c-f5ba-47d1-aae1-5baf948384b5\" (UID: \"7e3aac8c-f5ba-47d1-aae1-5baf948384b5\") " Nov 02 22:28:58.214491 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:58.210668 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-openshift-service-ca" pod "7e3aac8c-f5ba-47d1-aae1-5baf948384b5" (UID: "7e3aac8c-f5ba-47d1-aae1-5baf948384b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:28:58.214491 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:58.213051 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-maas-api-tls" pod "7e3aac8c-f5ba-47d1-aae1-5baf948384b5" (UID: "7e3aac8c-f5ba-47d1-aae1-5baf948384b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:28:58.214491 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:58.213402 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-kube-api-access-cwsdv" pod "7e3aac8c-f5ba-47d1-aae1-5baf948384b5" (UID: "7e3aac8c-f5ba-47d1-aae1-5baf948384b5"). InnerVolumeSpecName "kube-api-access-cwsdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:28:58.217333 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:58.217303 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-metrics-certs" pod "7e3aac8c-f5ba-47d1-aae1-5baf948384b5" (UID: "7e3aac8c-f5ba-47d1-aae1-5baf948384b5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:28:58.311646 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:58.311572 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:58.311646 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:58.311600 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:58.311646 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:58.311611 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:58.311646 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:58.311619 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwsdv\" (UniqueName: \"kubernetes.io/projected/7e3aac8c-f5ba-47d1-aae1-5baf948384b5-kube-api-access-cwsdv\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:58.469912 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:28:58.469873 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:28:58.561889 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:58.561816 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt"] Nov 02 22:28:58.566177 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:58.566149 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cf2a53-f5cbb89b5-ck5bt"] Nov 02 22:28:59.029527 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:59.029471 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:59.304346 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:59.304242 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:59.777376 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:59.777343 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e3aac8c-f5ba-47d1-aae1-5baf948384b5" path="/var/lib/kubelet/pods/7e3aac8c-f5ba-47d1-aae1-5baf948384b5/volumes" Nov 02 22:28:59.879884 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:28:59.879845 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:00.344497 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:00.344461 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:00.563232 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:00.563197 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:00.782147 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:00.782109 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:01.136008 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:01.135912 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:01.600396 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:01.600350 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:02.034124 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:02.034084 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:02.588102 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:02.588066 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:02.967472 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:02.967370 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:05.459267 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:05.459228 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:06.478194 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:06.478159 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j"] Nov 02 22:29:06.525663 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:06.525631 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9"] Nov 02 22:29:06.676089 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:06.676056 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:07.849065 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:07.849026 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx"] Nov 02 22:29:08.009983 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:08.009944 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:10.701414 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:10.701374 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:12.830638 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:12.830598 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:14.170349 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:14.170316 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:14.918830 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:14.918787 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft"] Nov 02 22:29:14.919161 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:14.919113 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft" podUID="dde2493b-74dc-411e-a991-cb7f91bda173" containerName="maas-api" containerID="cri-o://215581e968ce16830fdc98ece279bb647a6a028e9750c5fb249756b0c1450032" gracePeriod=30 Nov 02 22:29:15.111450 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:15.111412 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:15.201711 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:15.201682 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft" Nov 02 22:29:15.259208 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:15.259170 4935 generic.go:356] "Generic (PLEG): container finished" podID="dde2493b-74dc-411e-a991-cb7f91bda173" containerID="215581e968ce16830fdc98ece279bb647a6a028e9750c5fb249756b0c1450032" exitCode=0 Nov 02 22:29:15.259208 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:15.259214 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft" Nov 02 22:29:15.259437 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:15.259254 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft" event={"ID":"dde2493b-74dc-411e-a991-cb7f91bda173","Type":"ContainerDied","Data":"215581e968ce16830fdc98ece279bb647a6a028e9750c5fb249756b0c1450032"} Nov 02 22:29:15.259437 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:15.259315 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft" event={"ID":"dde2493b-74dc-411e-a991-cb7f91bda173","Type":"ContainerDied","Data":"f1d74f7f1f38f841db4009b15951c28f3156f172fbddb0f99ec210a2cee59483"} Nov 02 22:29:15.259437 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:15.259332 4935 scope.go:122] "RemoveContainer" containerID="215581e968ce16830fdc98ece279bb647a6a028e9750c5fb249756b0c1450032" Nov 02 22:29:15.269907 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:15.269886 4935 scope.go:122] "RemoveContainer" containerID="215581e968ce16830fdc98ece279bb647a6a028e9750c5fb249756b0c1450032" Nov 02 22:29:15.270256 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:29:15.270230 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"215581e968ce16830fdc98ece279bb647a6a028e9750c5fb249756b0c1450032\": container with ID starting with 215581e968ce16830fdc98ece279bb647a6a028e9750c5fb249756b0c1450032 not found: ID does not exist" containerID="215581e968ce16830fdc98ece279bb647a6a028e9750c5fb249756b0c1450032" Nov 02 22:29:15.270325 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:15.270270 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"215581e968ce16830fdc98ece279bb647a6a028e9750c5fb249756b0c1450032"} err="failed to get container status \"215581e968ce16830fdc98ece279bb647a6a028e9750c5fb249756b0c1450032\": rpc error: code = NotFound desc = could not find container \"215581e968ce16830fdc98ece279bb647a6a028e9750c5fb249756b0c1450032\": container with ID starting with 215581e968ce16830fdc98ece279bb647a6a028e9750c5fb249756b0c1450032 not found: ID does not exist" Nov 02 22:29:15.299179 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:15.299114 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dde2493b-74dc-411e-a991-cb7f91bda173-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dde2493b-74dc-411e-a991-cb7f91bda173-openshift-service-ca\") pod \"dde2493b-74dc-411e-a991-cb7f91bda173\" (UID: \"dde2493b-74dc-411e-a991-cb7f91bda173\") " Nov 02 22:29:15.299346 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:15.299270 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dde2493b-74dc-411e-a991-cb7f91bda173-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dde2493b-74dc-411e-a991-cb7f91bda173-maas-api-tls\") pod \"dde2493b-74dc-411e-a991-cb7f91bda173\" (UID: \"dde2493b-74dc-411e-a991-cb7f91bda173\") " Nov 02 22:29:15.299346 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:15.299299 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dde2493b-74dc-411e-a991-cb7f91bda173-kube-api-access-jbpp8\" (UniqueName: \"kubernetes.io/projected/dde2493b-74dc-411e-a991-cb7f91bda173-kube-api-access-jbpp8\") pod \"dde2493b-74dc-411e-a991-cb7f91bda173\" (UID: \"dde2493b-74dc-411e-a991-cb7f91bda173\") " Nov 02 22:29:15.299464 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:15.299349 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dde2493b-74dc-411e-a991-cb7f91bda173-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dde2493b-74dc-411e-a991-cb7f91bda173-metrics-certs\") pod \"dde2493b-74dc-411e-a991-cb7f91bda173\" (UID: \"dde2493b-74dc-411e-a991-cb7f91bda173\") " Nov 02 22:29:15.299524 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:15.299457 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dde2493b-74dc-411e-a991-cb7f91bda173-openshift-service-ca" pod "dde2493b-74dc-411e-a991-cb7f91bda173" (UID: "dde2493b-74dc-411e-a991-cb7f91bda173"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:29:15.300653 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:15.300623 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dde2493b-74dc-411e-a991-cb7f91bda173-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:15.302113 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:15.302072 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dde2493b-74dc-411e-a991-cb7f91bda173-metrics-certs" pod "dde2493b-74dc-411e-a991-cb7f91bda173" (UID: "dde2493b-74dc-411e-a991-cb7f91bda173"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:29:15.302113 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:15.302093 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dde2493b-74dc-411e-a991-cb7f91bda173-maas-api-tls" pod "dde2493b-74dc-411e-a991-cb7f91bda173" (UID: "dde2493b-74dc-411e-a991-cb7f91bda173"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:29:15.302349 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:15.302292 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dde2493b-74dc-411e-a991-cb7f91bda173-kube-api-access-jbpp8" pod "dde2493b-74dc-411e-a991-cb7f91bda173" (UID: "dde2493b-74dc-411e-a991-cb7f91bda173"). InnerVolumeSpecName "kube-api-access-jbpp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:15.402040 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:15.402005 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dde2493b-74dc-411e-a991-cb7f91bda173-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:15.402040 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:15.402039 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbpp8\" (UniqueName: \"kubernetes.io/projected/dde2493b-74dc-411e-a991-cb7f91bda173-kube-api-access-jbpp8\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:15.402310 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:15.402054 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dde2493b-74dc-411e-a991-cb7f91bda173-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:15.677490 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:15.677455 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft"] Nov 02 22:29:15.681487 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:15.681457 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-7cde84-794c548c75-5h8ft"] Nov 02 22:29:15.779165 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:15.779114 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dde2493b-74dc-411e-a991-cb7f91bda173" path="/var/lib/kubelet/pods/dde2493b-74dc-411e-a991-cb7f91bda173/volumes" Nov 02 22:29:16.926424 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:16.926388 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:17.165725 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:17.165646 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:17.381877 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:17.381828 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:18.487790 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:18.487761 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:18.689253 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:18.689219 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:18.894245 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:18.894205 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:19.144506 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:19.144373 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:19.351997 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:19.351963 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:19.909532 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:19.909408 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:20.229895 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:20.229811 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:20.714262 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:20.714223 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:20.915935 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:20.915897 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:21.489531 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:21.489497 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:21.499204 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:21.499174 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" podUID="7ba3a2e4-a47f-425a-8c65-5813ef0939c9" containerName="main" containerID="cri-o://e8ca277623431acab078d34790f59e15e624602b18d9857858383e4d3578ead6" gracePeriod=45 Nov 02 22:29:21.546246 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:21.546208 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" podUID="1d6a5790-dd1f-4721-bbde-3aa6b14999b8" containerName="main" containerID="cri-o://6b90bbe4ba8d433cc5a63e6a4e7b3aa6b3d3a0634fbe47cd783d79e51a780003" gracePeriod=45 Nov 02 22:29:21.591413 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:21.591375 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" podUID="1d6a5790-dd1f-4721-bbde-3aa6b14999b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": dial tcp 10.132.0.62:8000: connect: connection refused" Nov 02 22:29:21.691607 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:21.691555 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" podUID="7ba3a2e4-a47f-425a-8c65-5813ef0939c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": dial tcp 10.132.0.70:8000: connect: connection refused" Nov 02 22:29:21.977507 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:21.977334 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" Nov 02 22:29:21.980309 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:21.980284 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" Nov 02 22:29:22.109460 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.109421 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-model-cache\") pod \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\" (UID: \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\") " Nov 02 22:29:22.109635 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.109501 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-dshm\") pod \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\" (UID: \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\") " Nov 02 22:29:22.109635 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.109559 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-tmp-dir\") pod \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\" (UID: \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\") " Nov 02 22:29:22.109635 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.109593 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-tls-certs\") pod \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\" (UID: \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\") " Nov 02 22:29:22.109635 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.109630 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-tls-certs\" (UniqueName: \"kubernetes.io/secret/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-tls-certs\") pod \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\" (UID: \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\") " Nov 02 22:29:22.109839 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.109691 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-kube-api-access-2knmw\" (UniqueName: \"kubernetes.io/projected/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-kube-api-access-2knmw\") pod \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\" (UID: \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\") " Nov 02 22:29:22.109839 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.109723 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-dshm\") pod \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\" (UID: \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\") " Nov 02 22:29:22.109839 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.109820 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-tmp-dir\") pod \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\" (UID: \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\") " Nov 02 22:29:22.109985 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.109840 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-model-cache\") pod \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\" (UID: \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\") " Nov 02 22:29:22.109985 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.109864 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-home\" (UniqueName: \"kubernetes.io/empty-dir/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-home\") pod \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\" (UID: \"1d6a5790-dd1f-4721-bbde-3aa6b14999b8\") " Nov 02 22:29:22.109985 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.109900 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-home\" (UniqueName: \"kubernetes.io/empty-dir/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-home\") pod \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\" (UID: \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\") " Nov 02 22:29:22.109985 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.109930 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-kube-api-access-c7nxl\" (UniqueName: \"kubernetes.io/projected/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-kube-api-access-c7nxl\") pod \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\" (UID: \"7ba3a2e4-a47f-425a-8c65-5813ef0939c9\") " Nov 02 22:29:22.111859 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.111096 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-model-cache" pod "1d6a5790-dd1f-4721-bbde-3aa6b14999b8" (UID: "1d6a5790-dd1f-4721-bbde-3aa6b14999b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:22.111859 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.111385 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-tmp-dir" pod "1d6a5790-dd1f-4721-bbde-3aa6b14999b8" (UID: "1d6a5790-dd1f-4721-bbde-3aa6b14999b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:22.111859 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.111393 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-tmp-dir" pod "7ba3a2e4-a47f-425a-8c65-5813ef0939c9" (UID: "7ba3a2e4-a47f-425a-8c65-5813ef0939c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:22.111859 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.111611 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-home" pod "7ba3a2e4-a47f-425a-8c65-5813ef0939c9" (UID: "7ba3a2e4-a47f-425a-8c65-5813ef0939c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:22.111859 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.111625 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-model-cache" pod "7ba3a2e4-a47f-425a-8c65-5813ef0939c9" (UID: "7ba3a2e4-a47f-425a-8c65-5813ef0939c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:22.112552 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.112290 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-home" pod "1d6a5790-dd1f-4721-bbde-3aa6b14999b8" (UID: "1d6a5790-dd1f-4721-bbde-3aa6b14999b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:22.113290 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.113264 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-kube-api-access-c7nxl" pod "7ba3a2e4-a47f-425a-8c65-5813ef0939c9" (UID: "7ba3a2e4-a47f-425a-8c65-5813ef0939c9"). InnerVolumeSpecName "kube-api-access-c7nxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:22.113851 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.113829 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-dshm" pod "7ba3a2e4-a47f-425a-8c65-5813ef0939c9" (UID: "7ba3a2e4-a47f-425a-8c65-5813ef0939c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:22.113986 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.113921 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-tls-certs" pod "7ba3a2e4-a47f-425a-8c65-5813ef0939c9" (UID: "7ba3a2e4-a47f-425a-8c65-5813ef0939c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:29:22.114711 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.114684 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-tls-certs" pod "1d6a5790-dd1f-4721-bbde-3aa6b14999b8" (UID: "1d6a5790-dd1f-4721-bbde-3aa6b14999b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:29:22.115669 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.115643 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-dshm" pod "1d6a5790-dd1f-4721-bbde-3aa6b14999b8" (UID: "1d6a5790-dd1f-4721-bbde-3aa6b14999b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:22.116553 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.116527 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-kube-api-access-2knmw" pod "1d6a5790-dd1f-4721-bbde-3aa6b14999b8" (UID: "1d6a5790-dd1f-4721-bbde-3aa6b14999b8"). InnerVolumeSpecName "kube-api-access-2knmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:22.212107 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.212069 4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-tmp-dir\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:22.212107 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.212110 4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-model-cache\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:22.212314 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.212125 4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-home\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:22.212314 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.212158 4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-home\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:22.212314 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.212175 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7nxl\" (UniqueName: \"kubernetes.io/projected/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-kube-api-access-c7nxl\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:22.212314 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.212189 4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-model-cache\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:22.212314 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.212203 4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-dshm\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:22.212314 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.212216 4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-tmp-dir\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:22.212314 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.212229 4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ba3a2e4-a47f-425a-8c65-5813ef0939c9-tls-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:22.212314 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.212241 4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-tls-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:22.212314 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.212255 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2knmw\" (UniqueName: \"kubernetes.io/projected/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-kube-api-access-2knmw\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:22.212314 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.212268 4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d6a5790-dd1f-4721-bbde-3aa6b14999b8-dshm\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:22.305897 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.305862 4935 generic.go:356] "Generic (PLEG): container finished" podID="1d6a5790-dd1f-4721-bbde-3aa6b14999b8" containerID="6b90bbe4ba8d433cc5a63e6a4e7b3aa6b3d3a0634fbe47cd783d79e51a780003" exitCode=0 Nov 02 22:29:22.306096 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.305905 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" Nov 02 22:29:22.306096 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.305943 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" event={"ID":"1d6a5790-dd1f-4721-bbde-3aa6b14999b8","Type":"ContainerDied","Data":"6b90bbe4ba8d433cc5a63e6a4e7b3aa6b3d3a0634fbe47cd783d79e51a780003"} Nov 02 22:29:22.306096 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.305982 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9" event={"ID":"1d6a5790-dd1f-4721-bbde-3aa6b14999b8","Type":"ContainerDied","Data":"b0c217839ee5af527b55c676c5eee70dce2674788248b811fe44eecd41b43944"} Nov 02 22:29:22.306096 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.306000 4935 scope.go:122] "RemoveContainer" containerID="6b90bbe4ba8d433cc5a63e6a4e7b3aa6b3d3a0634fbe47cd783d79e51a780003" Nov 02 22:29:22.307436 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.307415 4935 generic.go:356] "Generic (PLEG): container finished" podID="7ba3a2e4-a47f-425a-8c65-5813ef0939c9" containerID="e8ca277623431acab078d34790f59e15e624602b18d9857858383e4d3578ead6" exitCode=0 Nov 02 22:29:22.307564 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.307447 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" Nov 02 22:29:22.307564 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.307496 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" event={"ID":"7ba3a2e4-a47f-425a-8c65-5813ef0939c9","Type":"ContainerDied","Data":"e8ca277623431acab078d34790f59e15e624602b18d9857858383e4d3578ead6"} Nov 02 22:29:22.307564 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.307542 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j" event={"ID":"7ba3a2e4-a47f-425a-8c65-5813ef0939c9","Type":"ContainerDied","Data":"4940b2e68f2a913f82557e55133542b7baea6d621df78494a1fe0e5d09289e00"} Nov 02 22:29:22.316088 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.316049 4935 scope.go:122] "RemoveContainer" containerID="6b90bbe4ba8d433cc5a63e6a4e7b3aa6b3d3a0634fbe47cd783d79e51a780003" Nov 02 22:29:22.316427 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:29:22.316400 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b90bbe4ba8d433cc5a63e6a4e7b3aa6b3d3a0634fbe47cd783d79e51a780003\": container with ID starting with 6b90bbe4ba8d433cc5a63e6a4e7b3aa6b3d3a0634fbe47cd783d79e51a780003 not found: ID does not exist" containerID="6b90bbe4ba8d433cc5a63e6a4e7b3aa6b3d3a0634fbe47cd783d79e51a780003" Nov 02 22:29:22.316516 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.316438 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b90bbe4ba8d433cc5a63e6a4e7b3aa6b3d3a0634fbe47cd783d79e51a780003"} err="failed to get container status \"6b90bbe4ba8d433cc5a63e6a4e7b3aa6b3d3a0634fbe47cd783d79e51a780003\": rpc error: code = NotFound desc = could not find container \"6b90bbe4ba8d433cc5a63e6a4e7b3aa6b3d3a0634fbe47cd783d79e51a780003\": container with ID starting with 6b90bbe4ba8d433cc5a63e6a4e7b3aa6b3d3a0634fbe47cd783d79e51a780003 not found: ID does not exist" Nov 02 22:29:22.316516 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.316463 4935 scope.go:122] "RemoveContainer" containerID="e8ca277623431acab078d34790f59e15e624602b18d9857858383e4d3578ead6" Nov 02 22:29:22.325500 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.325482 4935 scope.go:122] "RemoveContainer" containerID="e8ca277623431acab078d34790f59e15e624602b18d9857858383e4d3578ead6" Nov 02 22:29:22.325769 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:29:22.325749 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8ca277623431acab078d34790f59e15e624602b18d9857858383e4d3578ead6\": container with ID starting with e8ca277623431acab078d34790f59e15e624602b18d9857858383e4d3578ead6 not found: ID does not exist" containerID="e8ca277623431acab078d34790f59e15e624602b18d9857858383e4d3578ead6" Nov 02 22:29:22.325835 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.325779 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ca277623431acab078d34790f59e15e624602b18d9857858383e4d3578ead6"} err="failed to get container status \"e8ca277623431acab078d34790f59e15e624602b18d9857858383e4d3578ead6\": rpc error: code = NotFound desc = could not find container \"e8ca277623431acab078d34790f59e15e624602b18d9857858383e4d3578ead6\": container with ID starting with e8ca277623431acab078d34790f59e15e624602b18d9857858383e4d3578ead6 not found: ID does not exist" Nov 02 22:29:22.459191 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.459089 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j"] Nov 02 22:29:22.462186 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.462153 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-6fa533/distinct-2-w2-6fa533-kserve-7f6749b868-cc68j"] Nov 02 22:29:22.560192 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.560151 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9"] Nov 02 22:29:22.562649 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.562617 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:22.564200 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.564118 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-6fa533/simulator-w2-6fa533-kserve-658676577-55vz9"] Nov 02 22:29:22.876488 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.876431 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" podUID="39b34814-d9a5-4505-9906-cd18e55eed4b" containerName="main" containerID="cri-o://a3f3ac2bbdd411506337e19d11884b0c3780c7652f4be7ddcb28c55e89518075" gracePeriod=45 Nov 02 22:29:22.881005 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:22.880973 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" podUID="39b34814-d9a5-4505-9906-cd18e55eed4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": EOF" Nov 02 22:29:23.168119 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.168089 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" Nov 02 22:29:23.317792 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.317756 4935 generic.go:356] "Generic (PLEG): container finished" podID="39b34814-d9a5-4505-9906-cd18e55eed4b" containerID="a3f3ac2bbdd411506337e19d11884b0c3780c7652f4be7ddcb28c55e89518075" exitCode=0 Nov 02 22:29:23.318001 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.317893 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" event={"ID":"39b34814-d9a5-4505-9906-cd18e55eed4b","Type":"ContainerDied","Data":"a3f3ac2bbdd411506337e19d11884b0c3780c7652f4be7ddcb28c55e89518075"} Nov 02 22:29:23.318001 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.317930 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" event={"ID":"39b34814-d9a5-4505-9906-cd18e55eed4b","Type":"ContainerDied","Data":"d07688cd524991ccf332308b8d5c99e2b0ef97faec5363eea9bc0fd01c6ab289"} Nov 02 22:29:23.318001 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.317951 4935 scope.go:122] "RemoveContainer" containerID="a3f3ac2bbdd411506337e19d11884b0c3780c7652f4be7ddcb28c55e89518075" Nov 02 22:29:23.318119 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.318039 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx" Nov 02 22:29:23.339174 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.327868 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b34814-d9a5-4505-9906-cd18e55eed4b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39b34814-d9a5-4505-9906-cd18e55eed4b-model-cache\") pod \"39b34814-d9a5-4505-9906-cd18e55eed4b\" (UID: \"39b34814-d9a5-4505-9906-cd18e55eed4b\") " Nov 02 22:29:23.339174 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.327914 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39b34814-d9a5-4505-9906-cd18e55eed4b-tls-certs\" (UniqueName: \"kubernetes.io/secret/39b34814-d9a5-4505-9906-cd18e55eed4b-tls-certs\") pod \"39b34814-d9a5-4505-9906-cd18e55eed4b\" (UID: \"39b34814-d9a5-4505-9906-cd18e55eed4b\") " Nov 02 22:29:23.339174 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.328071 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39b34814-d9a5-4505-9906-cd18e55eed4b-kube-api-access-b2wvs\" (UniqueName: \"kubernetes.io/projected/39b34814-d9a5-4505-9906-cd18e55eed4b-kube-api-access-b2wvs\") pod \"39b34814-d9a5-4505-9906-cd18e55eed4b\" (UID: \"39b34814-d9a5-4505-9906-cd18e55eed4b\") " Nov 02 22:29:23.339174 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.328123 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b34814-d9a5-4505-9906-cd18e55eed4b-home\" (UniqueName: \"kubernetes.io/empty-dir/39b34814-d9a5-4505-9906-cd18e55eed4b-home\") pod \"39b34814-d9a5-4505-9906-cd18e55eed4b\" (UID: \"39b34814-d9a5-4505-9906-cd18e55eed4b\") " Nov 02 22:29:23.339174 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.328177 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b34814-d9a5-4505-9906-cd18e55eed4b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/39b34814-d9a5-4505-9906-cd18e55eed4b-dshm\") pod \"39b34814-d9a5-4505-9906-cd18e55eed4b\" (UID: \"39b34814-d9a5-4505-9906-cd18e55eed4b\") " Nov 02 22:29:23.339174 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.328235 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b34814-d9a5-4505-9906-cd18e55eed4b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b34814-d9a5-4505-9906-cd18e55eed4b-tmp-dir\") pod \"39b34814-d9a5-4505-9906-cd18e55eed4b\" (UID: \"39b34814-d9a5-4505-9906-cd18e55eed4b\") " Nov 02 22:29:23.339174 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.328256 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b34814-d9a5-4505-9906-cd18e55eed4b-model-cache" pod "39b34814-d9a5-4505-9906-cd18e55eed4b" (UID: "39b34814-d9a5-4505-9906-cd18e55eed4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:23.339174 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.328632 4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39b34814-d9a5-4505-9906-cd18e55eed4b-model-cache\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:23.339174 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.329011 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b34814-d9a5-4505-9906-cd18e55eed4b-home" pod "39b34814-d9a5-4505-9906-cd18e55eed4b" (UID: "39b34814-d9a5-4505-9906-cd18e55eed4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:23.339174 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.329051 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b34814-d9a5-4505-9906-cd18e55eed4b-tmp-dir" pod "39b34814-d9a5-4505-9906-cd18e55eed4b" (UID: "39b34814-d9a5-4505-9906-cd18e55eed4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:23.339174 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.330102 4935 scope.go:122] "RemoveContainer" containerID="a3f3ac2bbdd411506337e19d11884b0c3780c7652f4be7ddcb28c55e89518075" Nov 02 22:29:23.339174 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:29:23.330424 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3f3ac2bbdd411506337e19d11884b0c3780c7652f4be7ddcb28c55e89518075\": container with ID starting with a3f3ac2bbdd411506337e19d11884b0c3780c7652f4be7ddcb28c55e89518075 not found: ID does not exist" containerID="a3f3ac2bbdd411506337e19d11884b0c3780c7652f4be7ddcb28c55e89518075" Nov 02 22:29:23.339174 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.330457 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3f3ac2bbdd411506337e19d11884b0c3780c7652f4be7ddcb28c55e89518075"} err="failed to get container status \"a3f3ac2bbdd411506337e19d11884b0c3780c7652f4be7ddcb28c55e89518075\": rpc error: code = NotFound desc = could not find container \"a3f3ac2bbdd411506337e19d11884b0c3780c7652f4be7ddcb28c55e89518075\": container with ID starting with a3f3ac2bbdd411506337e19d11884b0c3780c7652f4be7ddcb28c55e89518075 not found: ID does not exist" Nov 02 22:29:23.339174 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.331759 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b34814-d9a5-4505-9906-cd18e55eed4b-kube-api-access-b2wvs" pod "39b34814-d9a5-4505-9906-cd18e55eed4b" (UID: "39b34814-d9a5-4505-9906-cd18e55eed4b"). InnerVolumeSpecName "kube-api-access-b2wvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:23.339174 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.336268 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39b34814-d9a5-4505-9906-cd18e55eed4b-tls-certs" pod "39b34814-d9a5-4505-9906-cd18e55eed4b" (UID: "39b34814-d9a5-4505-9906-cd18e55eed4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:29:23.342335 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.342307 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b34814-d9a5-4505-9906-cd18e55eed4b-dshm" pod "39b34814-d9a5-4505-9906-cd18e55eed4b" (UID: "39b34814-d9a5-4505-9906-cd18e55eed4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:23.429928 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.429843 4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39b34814-d9a5-4505-9906-cd18e55eed4b-dshm\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:23.429928 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.429873 4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b34814-d9a5-4505-9906-cd18e55eed4b-tmp-dir\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:23.429928 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.429882 4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b34814-d9a5-4505-9906-cd18e55eed4b-tls-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:23.429928 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.429891 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2wvs\" (UniqueName: \"kubernetes.io/projected/39b34814-d9a5-4505-9906-cd18e55eed4b-kube-api-access-b2wvs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:23.429928 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.429900 4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39b34814-d9a5-4505-9906-cd18e55eed4b-home\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:23.733718 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.733626 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx"] Nov 02 22:29:23.786887 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.786839 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d6a5790-dd1f-4721-bbde-3aa6b14999b8" path="/var/lib/kubelet/pods/1d6a5790-dd1f-4721-bbde-3aa6b14999b8/volumes" Nov 02 22:29:23.787425 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.787400 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ba3a2e4-a47f-425a-8c65-5813ef0939c9" path="/var/lib/kubelet/pods/7ba3a2e4-a47f-425a-8c65-5813ef0939c9/volumes" Nov 02 22:29:23.787755 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:23.787736 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5f765a/distinct-w1-5f765a-kserve-6f566d44d-wcvrx"] Nov 02 22:29:25.020904 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:25.020866 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:25.777605 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:25.777567 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39b34814-d9a5-4505-9906-cd18e55eed4b" path="/var/lib/kubelet/pods/39b34814-d9a5-4505-9906-cd18e55eed4b/volumes" Nov 02 22:29:26.169034 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:26.168942 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:26.384030 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:26.383997 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:26.711744 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:26.711709 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:26.919901 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:26.919860 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:28.012870 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:28.012836 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:28.470021 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:29:28.469937 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:29:32.092471 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:32.092435 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:32.291952 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:32.291902 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:32.605258 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:32.605222 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr"] Nov 02 22:29:32.605493 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:32.605468 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr" podUID="1644e4fb-94a2-414c-9c35-99abbc8fe2b2" containerName="maas-api" containerID="cri-o://4c9483933b04bf697513a1efc9f280acb08ab35ebef40e49bc82748791750736" gracePeriod=30 Nov 02 22:29:32.785210 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:32.782312 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:32.910244 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:32.910209 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr" Nov 02 22:29:33.037158 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:33.033543 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1644e4fb-94a2-414c-9c35-99abbc8fe2b2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1644e4fb-94a2-414c-9c35-99abbc8fe2b2-openshift-service-ca\") pod \"1644e4fb-94a2-414c-9c35-99abbc8fe2b2\" (UID: \"1644e4fb-94a2-414c-9c35-99abbc8fe2b2\") " Nov 02 22:29:33.049166 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:33.033932 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1644e4fb-94a2-414c-9c35-99abbc8fe2b2-openshift-service-ca" pod "1644e4fb-94a2-414c-9c35-99abbc8fe2b2" (UID: "1644e4fb-94a2-414c-9c35-99abbc8fe2b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:29:33.049166 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:33.045603 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1644e4fb-94a2-414c-9c35-99abbc8fe2b2-kube-api-access-4hqr6\" (UniqueName: \"kubernetes.io/projected/1644e4fb-94a2-414c-9c35-99abbc8fe2b2-kube-api-access-4hqr6\") pod \"1644e4fb-94a2-414c-9c35-99abbc8fe2b2\" (UID: \"1644e4fb-94a2-414c-9c35-99abbc8fe2b2\") " Nov 02 22:29:33.049166 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:33.045661 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1644e4fb-94a2-414c-9c35-99abbc8fe2b2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1644e4fb-94a2-414c-9c35-99abbc8fe2b2-maas-api-tls\") pod \"1644e4fb-94a2-414c-9c35-99abbc8fe2b2\" (UID: \"1644e4fb-94a2-414c-9c35-99abbc8fe2b2\") " Nov 02 22:29:33.049166 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:33.045692 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1644e4fb-94a2-414c-9c35-99abbc8fe2b2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1644e4fb-94a2-414c-9c35-99abbc8fe2b2-metrics-certs\") pod \"1644e4fb-94a2-414c-9c35-99abbc8fe2b2\" (UID: \"1644e4fb-94a2-414c-9c35-99abbc8fe2b2\") " Nov 02 22:29:33.049166 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:33.046224 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1644e4fb-94a2-414c-9c35-99abbc8fe2b2-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:33.049166 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:33.048796 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1644e4fb-94a2-414c-9c35-99abbc8fe2b2-kube-api-access-4hqr6" pod "1644e4fb-94a2-414c-9c35-99abbc8fe2b2" (UID: "1644e4fb-94a2-414c-9c35-99abbc8fe2b2"). InnerVolumeSpecName "kube-api-access-4hqr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:33.050768 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:33.050730 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1644e4fb-94a2-414c-9c35-99abbc8fe2b2-maas-api-tls" pod "1644e4fb-94a2-414c-9c35-99abbc8fe2b2" (UID: "1644e4fb-94a2-414c-9c35-99abbc8fe2b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:29:33.060417 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:33.060368 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1644e4fb-94a2-414c-9c35-99abbc8fe2b2-metrics-certs" pod "1644e4fb-94a2-414c-9c35-99abbc8fe2b2" (UID: "1644e4fb-94a2-414c-9c35-99abbc8fe2b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:29:33.147445 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:33.147353 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hqr6\" (UniqueName: \"kubernetes.io/projected/1644e4fb-94a2-414c-9c35-99abbc8fe2b2-kube-api-access-4hqr6\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:33.147445 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:33.147387 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1644e4fb-94a2-414c-9c35-99abbc8fe2b2-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:33.147445 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:33.147402 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1644e4fb-94a2-414c-9c35-99abbc8fe2b2-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:33.385735 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:33.385694 4935 generic.go:356] "Generic (PLEG): container finished" podID="1644e4fb-94a2-414c-9c35-99abbc8fe2b2" containerID="4c9483933b04bf697513a1efc9f280acb08ab35ebef40e49bc82748791750736" exitCode=0 Nov 02 22:29:33.385889 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:33.385737 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr" Nov 02 22:29:33.385889 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:33.385770 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr" event={"ID":"1644e4fb-94a2-414c-9c35-99abbc8fe2b2","Type":"ContainerDied","Data":"4c9483933b04bf697513a1efc9f280acb08ab35ebef40e49bc82748791750736"} Nov 02 22:29:33.385889 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:33.385823 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr" event={"ID":"1644e4fb-94a2-414c-9c35-99abbc8fe2b2","Type":"ContainerDied","Data":"5c5a5f8d2f937d3fc2bedcfb9dc980a13973422855b86beed0b33a926ce5cf88"} Nov 02 22:29:33.385889 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:33.385846 4935 scope.go:122] "RemoveContainer" containerID="4c9483933b04bf697513a1efc9f280acb08ab35ebef40e49bc82748791750736" Nov 02 22:29:33.396545 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:33.396526 4935 scope.go:122] "RemoveContainer" containerID="4c9483933b04bf697513a1efc9f280acb08ab35ebef40e49bc82748791750736" Nov 02 22:29:33.396826 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:29:33.396804 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c9483933b04bf697513a1efc9f280acb08ab35ebef40e49bc82748791750736\": container with ID starting with 4c9483933b04bf697513a1efc9f280acb08ab35ebef40e49bc82748791750736 not found: ID does not exist" containerID="4c9483933b04bf697513a1efc9f280acb08ab35ebef40e49bc82748791750736" Nov 02 22:29:33.396908 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:33.396834 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c9483933b04bf697513a1efc9f280acb08ab35ebef40e49bc82748791750736"} err="failed to get container status \"4c9483933b04bf697513a1efc9f280acb08ab35ebef40e49bc82748791750736\": rpc error: code = NotFound desc = could not find container \"4c9483933b04bf697513a1efc9f280acb08ab35ebef40e49bc82748791750736\": container with ID starting with 4c9483933b04bf697513a1efc9f280acb08ab35ebef40e49bc82748791750736 not found: ID does not exist" Nov 02 22:29:33.538615 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:33.538582 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr"] Nov 02 22:29:33.544223 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:33.544196 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-58ca7f-7dfbcf746c-f2vzr"] Nov 02 22:29:33.780290 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:33.780244 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1644e4fb-94a2-414c-9c35-99abbc8fe2b2" path="/var/lib/kubelet/pods/1644e4fb-94a2-414c-9c35-99abbc8fe2b2/volumes" Nov 02 22:29:33.809452 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:33.809378 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws"] Nov 02 22:29:33.809609 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:33.809587 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws" podUID="84fe42ff-d1d9-4f93-82a3-418ce33034df" containerName="maas-api" containerID="cri-o://c9d468cdd8638d3c8dba604a3e7a33845cbde12333864b917e096187d37dd46d" gracePeriod=30 Nov 02 22:29:33.985986 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:33.985947 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:34.110385 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:34.110356 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws" Nov 02 22:29:34.260422 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:34.260384 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84fe42ff-d1d9-4f93-82a3-418ce33034df-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fe42ff-d1d9-4f93-82a3-418ce33034df-openshift-service-ca\") pod \"84fe42ff-d1d9-4f93-82a3-418ce33034df\" (UID: \"84fe42ff-d1d9-4f93-82a3-418ce33034df\") " Nov 02 22:29:34.260940 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:34.260460 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84fe42ff-d1d9-4f93-82a3-418ce33034df-metrics-certs\" (UniqueName: \"kubernetes.io/secret/84fe42ff-d1d9-4f93-82a3-418ce33034df-metrics-certs\") pod \"84fe42ff-d1d9-4f93-82a3-418ce33034df\" (UID: \"84fe42ff-d1d9-4f93-82a3-418ce33034df\") " Nov 02 22:29:34.260940 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:34.260669 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84fe42ff-d1d9-4f93-82a3-418ce33034df-kube-api-access-2hrr7\" (UniqueName: \"kubernetes.io/projected/84fe42ff-d1d9-4f93-82a3-418ce33034df-kube-api-access-2hrr7\") pod \"84fe42ff-d1d9-4f93-82a3-418ce33034df\" (UID: \"84fe42ff-d1d9-4f93-82a3-418ce33034df\") " Nov 02 22:29:34.260940 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:34.260709 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84fe42ff-d1d9-4f93-82a3-418ce33034df-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84fe42ff-d1d9-4f93-82a3-418ce33034df-maas-api-tls\") pod \"84fe42ff-d1d9-4f93-82a3-418ce33034df\" (UID: \"84fe42ff-d1d9-4f93-82a3-418ce33034df\") " Nov 02 22:29:34.261546 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:34.261514 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84fe42ff-d1d9-4f93-82a3-418ce33034df-openshift-service-ca" pod "84fe42ff-d1d9-4f93-82a3-418ce33034df" (UID: "84fe42ff-d1d9-4f93-82a3-418ce33034df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:29:34.263435 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:34.263366 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84fe42ff-d1d9-4f93-82a3-418ce33034df-kube-api-access-2hrr7" pod "84fe42ff-d1d9-4f93-82a3-418ce33034df" (UID: "84fe42ff-d1d9-4f93-82a3-418ce33034df"). InnerVolumeSpecName "kube-api-access-2hrr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:34.263927 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:34.263903 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84fe42ff-d1d9-4f93-82a3-418ce33034df-metrics-certs" pod "84fe42ff-d1d9-4f93-82a3-418ce33034df" (UID: "84fe42ff-d1d9-4f93-82a3-418ce33034df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:29:34.264168 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:34.264110 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84fe42ff-d1d9-4f93-82a3-418ce33034df-maas-api-tls" pod "84fe42ff-d1d9-4f93-82a3-418ce33034df" (UID: "84fe42ff-d1d9-4f93-82a3-418ce33034df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:29:34.361779 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:34.361680 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hrr7\" (UniqueName: \"kubernetes.io/projected/84fe42ff-d1d9-4f93-82a3-418ce33034df-kube-api-access-2hrr7\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:34.361779 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:34.361713 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84fe42ff-d1d9-4f93-82a3-418ce33034df-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:34.361779 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:34.361725 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fe42ff-d1d9-4f93-82a3-418ce33034df-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:34.361779 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:34.361739 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84fe42ff-d1d9-4f93-82a3-418ce33034df-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:34.392835 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:34.392796 4935 generic.go:356] "Generic (PLEG): container finished" podID="84fe42ff-d1d9-4f93-82a3-418ce33034df" containerID="c9d468cdd8638d3c8dba604a3e7a33845cbde12333864b917e096187d37dd46d" exitCode=0 Nov 02 22:29:34.393002 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:34.392846 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws" Nov 02 22:29:34.393002 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:34.392877 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws" event={"ID":"84fe42ff-d1d9-4f93-82a3-418ce33034df","Type":"ContainerDied","Data":"c9d468cdd8638d3c8dba604a3e7a33845cbde12333864b917e096187d37dd46d"} Nov 02 22:29:34.393002 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:34.392934 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws" event={"ID":"84fe42ff-d1d9-4f93-82a3-418ce33034df","Type":"ContainerDied","Data":"02131d28618671602bf817e015e8516f9c5cce0313cd0f44b00461c8e6d0ab48"} Nov 02 22:29:34.393002 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:34.392957 4935 scope.go:122] "RemoveContainer" containerID="c9d468cdd8638d3c8dba604a3e7a33845cbde12333864b917e096187d37dd46d" Nov 02 22:29:34.403946 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:34.403927 4935 scope.go:122] "RemoveContainer" containerID="c9d468cdd8638d3c8dba604a3e7a33845cbde12333864b917e096187d37dd46d" Nov 02 22:29:34.404219 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:29:34.404196 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d468cdd8638d3c8dba604a3e7a33845cbde12333864b917e096187d37dd46d\": container with ID starting with c9d468cdd8638d3c8dba604a3e7a33845cbde12333864b917e096187d37dd46d not found: ID does not exist" containerID="c9d468cdd8638d3c8dba604a3e7a33845cbde12333864b917e096187d37dd46d" Nov 02 22:29:34.404284 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:34.404226 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d468cdd8638d3c8dba604a3e7a33845cbde12333864b917e096187d37dd46d"} err="failed to get container status \"c9d468cdd8638d3c8dba604a3e7a33845cbde12333864b917e096187d37dd46d\": rpc error: code = NotFound desc = could not find container \"c9d468cdd8638d3c8dba604a3e7a33845cbde12333864b917e096187d37dd46d\": container with ID starting with c9d468cdd8638d3c8dba604a3e7a33845cbde12333864b917e096187d37dd46d not found: ID does not exist" Nov 02 22:29:34.548900 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:34.548865 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws"] Nov 02 22:29:34.553164 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:34.553126 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6fa533-6bb9655d8f-brxws"] Nov 02 22:29:35.776701 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:35.776667 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84fe42ff-d1d9-4f93-82a3-418ce33034df" path="/var/lib/kubelet/pods/84fe42ff-d1d9-4f93-82a3-418ce33034df/volumes" Nov 02 22:29:36.214839 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:36.214744 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:37.013820 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:37.013780 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:37.828716 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:37.828680 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:38.041509 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:38.041474 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:38.349822 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:38.349780 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:38.566902 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:38.566866 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:39.247911 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:39.247866 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:39.452920 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:39.452881 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:39.844414 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:39.844377 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:40.050288 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:40.050255 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:40.278940 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:40.278901 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:42.523435 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:42.523400 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:42.746737 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:42.746666 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:42.954904 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:42.954817 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:43.165536 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:43.165497 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:44.224263 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:44.224227 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:44.659673 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:44.659632 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:45.085715 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:45.085669 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:45.192074 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:45.192029 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt"] Nov 02 22:29:45.239099 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:45.239058 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm"] Nov 02 22:29:45.315159 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:45.315115 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:45.527709 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:45.527676 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:45.777083 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:45.777045 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:46.106260 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:46.106228 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:47.414937 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:47.414898 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:47.792435 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:47.792399 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k"] Nov 02 22:29:47.792635 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:47.792605 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" podUID="c5e1984c-0e41-4b67-b36d-7796b3267a76" containerName="istio-proxy" containerID="cri-o://50dc4b37692cad6a49f984a128151fa1ed65b562774850b721836c8f29765ff1" gracePeriod=30 Nov 02 22:29:47.852218 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:47.847713 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:48.066410 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:48.066332 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:48.780545 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:48.780508 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:49.250707 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:49.250622 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:50.406241 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:50.406207 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:51.499215 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:51.499169 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:51.929023 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:51.928986 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:53.155417 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.155389 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 22:29:53.309380 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.309337 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-istio-envoy\") pod \"c5e1984c-0e41-4b67-b36d-7796b3267a76\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " Nov 02 22:29:53.309582 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.309402 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-workload-certs\") pod \"c5e1984c-0e41-4b67-b36d-7796b3267a76\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " Nov 02 22:29:53.309582 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.309488 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-credential-socket\") pod \"c5e1984c-0e41-4b67-b36d-7796b3267a76\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " Nov 02 22:29:53.309582 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.309533 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5e1984c-0e41-4b67-b36d-7796b3267a76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e1984c-0e41-4b67-b36d-7796b3267a76-istiod-ca-cert\") pod \"c5e1984c-0e41-4b67-b36d-7796b3267a76\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " Nov 02 22:29:53.309752 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.309595 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-istio-data\") pod \"c5e1984c-0e41-4b67-b36d-7796b3267a76\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " Nov 02 22:29:53.309752 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.309619 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-workload-socket\") pod \"c5e1984c-0e41-4b67-b36d-7796b3267a76\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " Nov 02 22:29:53.309868 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.309773 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-workload-certs" pod "c5e1984c-0e41-4b67-b36d-7796b3267a76" (UID: "c5e1984c-0e41-4b67-b36d-7796b3267a76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:53.309868 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.309786 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-credential-socket" pod "c5e1984c-0e41-4b67-b36d-7796b3267a76" (UID: "c5e1984c-0e41-4b67-b36d-7796b3267a76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:53.309868 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.309834 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5e1984c-0e41-4b67-b36d-7796b3267a76-kube-api-access-jdjfx\" (UniqueName: \"kubernetes.io/projected/c5e1984c-0e41-4b67-b36d-7796b3267a76-kube-api-access-jdjfx\") pod \"c5e1984c-0e41-4b67-b36d-7796b3267a76\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " Nov 02 22:29:53.310079 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.309866 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-workload-socket" pod "c5e1984c-0e41-4b67-b36d-7796b3267a76" (UID: "c5e1984c-0e41-4b67-b36d-7796b3267a76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:53.310079 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.309880 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-istio-data" pod "c5e1984c-0e41-4b67-b36d-7796b3267a76" (UID: "c5e1984c-0e41-4b67-b36d-7796b3267a76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:53.310079 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.309913 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5e1984c-0e41-4b67-b36d-7796b3267a76-istio-token\" (UniqueName: \"kubernetes.io/projected/c5e1984c-0e41-4b67-b36d-7796b3267a76-istio-token\") pod \"c5e1984c-0e41-4b67-b36d-7796b3267a76\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " Nov 02 22:29:53.310079 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.309971 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5e1984c-0e41-4b67-b36d-7796b3267a76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e1984c-0e41-4b67-b36d-7796b3267a76-istio-podinfo\") pod \"c5e1984c-0e41-4b67-b36d-7796b3267a76\" (UID: \"c5e1984c-0e41-4b67-b36d-7796b3267a76\") " Nov 02 22:29:53.310079 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.310001 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e1984c-0e41-4b67-b36d-7796b3267a76-istiod-ca-cert" pod "c5e1984c-0e41-4b67-b36d-7796b3267a76" (UID: "c5e1984c-0e41-4b67-b36d-7796b3267a76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:29:53.310524 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.310468 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:53.310524 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.310501 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e1984c-0e41-4b67-b36d-7796b3267a76-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:53.310524 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.310517 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:53.310660 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.310532 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:53.310660 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.310548 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:53.314638 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.314603 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e1984c-0e41-4b67-b36d-7796b3267a76-istio-token" pod "c5e1984c-0e41-4b67-b36d-7796b3267a76" (UID: "c5e1984c-0e41-4b67-b36d-7796b3267a76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:53.314792 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.314766 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e1984c-0e41-4b67-b36d-7796b3267a76-kube-api-access-jdjfx" pod "c5e1984c-0e41-4b67-b36d-7796b3267a76" (UID: "c5e1984c-0e41-4b67-b36d-7796b3267a76"). InnerVolumeSpecName "kube-api-access-jdjfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:53.315336 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.315312 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-istio-envoy" pod "c5e1984c-0e41-4b67-b36d-7796b3267a76" (UID: "c5e1984c-0e41-4b67-b36d-7796b3267a76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:29:53.315985 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.315952 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5e1984c-0e41-4b67-b36d-7796b3267a76-istio-podinfo" pod "c5e1984c-0e41-4b67-b36d-7796b3267a76" (UID: "c5e1984c-0e41-4b67-b36d-7796b3267a76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:29:53.411628 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.411589 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdjfx\" (UniqueName: \"kubernetes.io/projected/c5e1984c-0e41-4b67-b36d-7796b3267a76-kube-api-access-jdjfx\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:53.411628 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.411624 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e1984c-0e41-4b67-b36d-7796b3267a76-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:53.411811 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.411640 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e1984c-0e41-4b67-b36d-7796b3267a76-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:53.411811 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.411652 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e1984c-0e41-4b67-b36d-7796b3267a76-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:53.526367 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.526330 4935 generic.go:356] "Generic (PLEG): container finished" podID="c5e1984c-0e41-4b67-b36d-7796b3267a76" containerID="50dc4b37692cad6a49f984a128151fa1ed65b562774850b721836c8f29765ff1" exitCode=0 Nov 02 22:29:53.526546 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.526399 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" Nov 02 22:29:53.526546 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.526413 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" event={"ID":"c5e1984c-0e41-4b67-b36d-7796b3267a76","Type":"ContainerDied","Data":"50dc4b37692cad6a49f984a128151fa1ed65b562774850b721836c8f29765ff1"} Nov 02 22:29:53.526546 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.526504 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k" event={"ID":"c5e1984c-0e41-4b67-b36d-7796b3267a76","Type":"ContainerDied","Data":"978e2be7527b1eefc5e8197d18de2650895f700b405a1744a226e2e250899460"} Nov 02 22:29:53.526546 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.526531 4935 scope.go:122] "RemoveContainer" containerID="50dc4b37692cad6a49f984a128151fa1ed65b562774850b721836c8f29765ff1" Nov 02 22:29:53.539379 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.539353 4935 scope.go:122] "RemoveContainer" containerID="50dc4b37692cad6a49f984a128151fa1ed65b562774850b721836c8f29765ff1" Nov 02 22:29:53.539738 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:29:53.539714 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50dc4b37692cad6a49f984a128151fa1ed65b562774850b721836c8f29765ff1\": container with ID starting with 50dc4b37692cad6a49f984a128151fa1ed65b562774850b721836c8f29765ff1 not found: ID does not exist" containerID="50dc4b37692cad6a49f984a128151fa1ed65b562774850b721836c8f29765ff1" Nov 02 22:29:53.539853 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.539744 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50dc4b37692cad6a49f984a128151fa1ed65b562774850b721836c8f29765ff1"} err="failed to get container status \"50dc4b37692cad6a49f984a128151fa1ed65b562774850b721836c8f29765ff1\": rpc error: code = NotFound desc = could not find container \"50dc4b37692cad6a49f984a128151fa1ed65b562774850b721836c8f29765ff1\": container with ID starting with 50dc4b37692cad6a49f984a128151fa1ed65b562774850b721836c8f29765ff1 not found: ID does not exist" Nov 02 22:29:53.561902 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.561832 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:53.680961 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.680930 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k"] Nov 02 22:29:53.683970 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.683939 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6fa533-openshift-default-9bc476db9-hk87k"] Nov 02 22:29:53.775911 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:53.775878 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5e1984c-0e41-4b67-b36d-7796b3267a76" path="/var/lib/kubelet/pods/c5e1984c-0e41-4b67-b36d-7796b3267a76/volumes" Nov 02 22:29:57.776919 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:57.776889 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:58.468800 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:29:58.468762 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:29:58.782454 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:58.782397 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:58.990396 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:58.990356 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:59.196043 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:29:59.195938 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:00.186756 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.186724 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k"] Nov 02 22:30:00.187202 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187186 4935 cpu_manager.go:414] "containerMap: removing container" podUID="84fe42ff-d1d9-4f93-82a3-418ce33034df" containerName="maas-api" Nov 02 22:30:00.187251 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187203 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84fe42ff-d1d9-4f93-82a3-418ce33034df" containerName="maas-api" Nov 02 22:30:00.187251 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187216 4935 cpu_manager.go:414] "containerMap: removing container" podUID="4b2d5ed1-0979-476f-bc8e-dfe5099b5f49" containerName="main" Nov 02 22:30:00.187251 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187222 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b2d5ed1-0979-476f-bc8e-dfe5099b5f49" containerName="main" Nov 02 22:30:00.187251 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187234 4935 cpu_manager.go:414] "containerMap: removing container" podUID="93565534-5250-4709-8284-beb0aea24584" containerName="maas-api" Nov 02 22:30:00.187251 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187243 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93565534-5250-4709-8284-beb0aea24584" containerName="maas-api" Nov 02 22:30:00.187251 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187252 4935 cpu_manager.go:414] "containerMap: removing container" podUID="39b34814-d9a5-4505-9906-cd18e55eed4b" containerName="main" Nov 02 22:30:00.187443 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187258 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39b34814-d9a5-4505-9906-cd18e55eed4b" containerName="main" Nov 02 22:30:00.187443 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187268 4935 cpu_manager.go:414] "containerMap: removing container" podUID="c5e1984c-0e41-4b67-b36d-7796b3267a76" containerName="istio-proxy" Nov 02 22:30:00.187443 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187273 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5e1984c-0e41-4b67-b36d-7796b3267a76" containerName="istio-proxy" Nov 02 22:30:00.187443 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187280 4935 cpu_manager.go:414] "containerMap: removing container" podUID="7ba3a2e4-a47f-425a-8c65-5813ef0939c9" containerName="main" Nov 02 22:30:00.187443 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187284 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ba3a2e4-a47f-425a-8c65-5813ef0939c9" containerName="main" Nov 02 22:30:00.187443 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187293 4935 cpu_manager.go:414] "containerMap: removing container" podUID="2750bd81-031e-4b2c-bb07-a0ab009da8aa" containerName="istio-proxy" Nov 02 22:30:00.187443 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187298 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2750bd81-031e-4b2c-bb07-a0ab009da8aa" containerName="istio-proxy" Nov 02 22:30:00.187443 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187305 4935 cpu_manager.go:414] "containerMap: removing container" podUID="232da504-01ae-4fb4-86cb-43c8ca0f6a7a" containerName="istio-proxy" Nov 02 22:30:00.187443 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187310 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="232da504-01ae-4fb4-86cb-43c8ca0f6a7a" containerName="istio-proxy" Nov 02 22:30:00.187443 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187319 4935 cpu_manager.go:414] "containerMap: removing container" podUID="7e3aac8c-f5ba-47d1-aae1-5baf948384b5" containerName="maas-api" Nov 02 22:30:00.187443 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187323 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e3aac8c-f5ba-47d1-aae1-5baf948384b5" containerName="maas-api" Nov 02 22:30:00.187443 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187338 4935 cpu_manager.go:414] "containerMap: removing container" podUID="1644e4fb-94a2-414c-9c35-99abbc8fe2b2" containerName="maas-api" Nov 02 22:30:00.187443 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187344 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1644e4fb-94a2-414c-9c35-99abbc8fe2b2" containerName="maas-api" Nov 02 22:30:00.187443 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187351 4935 cpu_manager.go:414] "containerMap: removing container" podUID="1d6a5790-dd1f-4721-bbde-3aa6b14999b8" containerName="main" Nov 02 22:30:00.187443 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187356 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d6a5790-dd1f-4721-bbde-3aa6b14999b8" containerName="main" Nov 02 22:30:00.187443 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187363 4935 cpu_manager.go:414] "containerMap: removing container" podUID="a45b8c7b-0efd-414f-a246-0b8ecb6349e3" containerName="main" Nov 02 22:30:00.187443 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187368 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a45b8c7b-0efd-414f-a246-0b8ecb6349e3" containerName="main" Nov 02 22:30:00.187443 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187375 4935 cpu_manager.go:414] "containerMap: removing container" podUID="fa69559b-d902-4486-92e3-32af344bf33f" containerName="maas-api" Nov 02 22:30:00.187443 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187379 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa69559b-d902-4486-92e3-32af344bf33f" containerName="maas-api" Nov 02 22:30:00.187443 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187387 4935 cpu_manager.go:414] "containerMap: removing container" podUID="dde2493b-74dc-411e-a991-cb7f91bda173" containerName="maas-api" Nov 02 22:30:00.187443 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187392 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dde2493b-74dc-411e-a991-cb7f91bda173" containerName="maas-api" Nov 02 22:30:00.188106 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187469 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="84fe42ff-d1d9-4f93-82a3-418ce33034df" containerName="maas-api" Nov 02 22:30:00.188106 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187476 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b2d5ed1-0979-476f-bc8e-dfe5099b5f49" containerName="main" Nov 02 22:30:00.188106 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187486 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="93565534-5250-4709-8284-beb0aea24584" containerName="maas-api" Nov 02 22:30:00.188106 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187492 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="39b34814-d9a5-4505-9906-cd18e55eed4b" containerName="main" Nov 02 22:30:00.188106 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187498 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5e1984c-0e41-4b67-b36d-7796b3267a76" containerName="istio-proxy" Nov 02 22:30:00.188106 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187503 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ba3a2e4-a47f-425a-8c65-5813ef0939c9" containerName="main" Nov 02 22:30:00.188106 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187508 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="2750bd81-031e-4b2c-bb07-a0ab009da8aa" containerName="istio-proxy" Nov 02 22:30:00.188106 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187513 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="232da504-01ae-4fb4-86cb-43c8ca0f6a7a" containerName="istio-proxy" Nov 02 22:30:00.188106 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187518 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e3aac8c-f5ba-47d1-aae1-5baf948384b5" containerName="maas-api" Nov 02 22:30:00.188106 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187526 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="1644e4fb-94a2-414c-9c35-99abbc8fe2b2" containerName="maas-api" Nov 02 22:30:00.188106 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187531 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d6a5790-dd1f-4721-bbde-3aa6b14999b8" containerName="main" Nov 02 22:30:00.188106 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187537 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="a45b8c7b-0efd-414f-a246-0b8ecb6349e3" containerName="main" Nov 02 22:30:00.188106 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187542 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa69559b-d902-4486-92e3-32af344bf33f" containerName="maas-api" Nov 02 22:30:00.188106 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.187547 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="dde2493b-74dc-411e-a991-cb7f91bda173" containerName="maas-api" Nov 02 22:30:00.192360 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.192334 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k" Nov 02 22:30:00.214507 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.214454 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" podUID="095af893-e150-43b9-b479-263a270d05c4" containerName="main" containerID="cri-o://c713f4ce3c43a590bdeeb5ba34c2e4f9516edcb8b56f22467e5c93a21d04b3b3" gracePeriod=45 Nov 02 22:30:00.262275 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.262242 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k"] Nov 02 22:30:00.276737 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.276698 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" podUID="3920de75-9c3e-4826-8de5-22fcc6d50b0e" containerName="main" containerID="cri-o://fb046d23c6efc411fd4cf323c6be57faf02a543f14805601fd86e5e409bbc374" gracePeriod=45 Nov 02 22:30:00.283871 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.283845 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd425cbe-3140-4b2c-86c4-60f5261795f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k\" (UID: \"cd425cbe-3140-4b2c-86c4-60f5261795f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k" Nov 02 22:30:00.283978 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.283878 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd6km\" (UniqueName: \"kubernetes.io/projected/cd425cbe-3140-4b2c-86c4-60f5261795f4-kube-api-access-vd6km\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k\" (UID: \"cd425cbe-3140-4b2c-86c4-60f5261795f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k" Nov 02 22:30:00.321043 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.321012 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wb4fk\"" Nov 02 22:30:00.361116 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.361079 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" podUID="3920de75-9c3e-4826-8de5-22fcc6d50b0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": dial tcp 10.132.0.60:8000: connect: connection refused" Nov 02 22:30:00.385161 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.385103 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd425cbe-3140-4b2c-86c4-60f5261795f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k\" (UID: \"cd425cbe-3140-4b2c-86c4-60f5261795f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k" Nov 02 22:30:00.385291 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.385175 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd6km\" (UniqueName: \"kubernetes.io/projected/cd425cbe-3140-4b2c-86c4-60f5261795f4-kube-api-access-vd6km\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k\" (UID: \"cd425cbe-3140-4b2c-86c4-60f5261795f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k" Nov 02 22:30:00.385826 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.385803 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd425cbe-3140-4b2c-86c4-60f5261795f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k\" (UID: \"cd425cbe-3140-4b2c-86c4-60f5261795f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k" Nov 02 22:30:00.438475 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.438394 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd6km\" (UniqueName: \"kubernetes.io/projected/cd425cbe-3140-4b2c-86c4-60f5261795f4-kube-api-access-vd6km\") pod \"maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k\" (UID: \"cd425cbe-3140-4b2c-86c4-60f5261795f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k" Nov 02 22:30:00.523544 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.523346 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k" Nov 02 22:30:00.552165 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.552109 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" Nov 02 22:30:00.568415 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.568391 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" Nov 02 22:30:00.571211 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.571180 4935 generic.go:356] "Generic (PLEG): container finished" podID="095af893-e150-43b9-b479-263a270d05c4" containerID="c713f4ce3c43a590bdeeb5ba34c2e4f9516edcb8b56f22467e5c93a21d04b3b3" exitCode=0 Nov 02 22:30:00.571344 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.571229 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" Nov 02 22:30:00.571344 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.571268 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" event={"ID":"095af893-e150-43b9-b479-263a270d05c4","Type":"ContainerDied","Data":"c713f4ce3c43a590bdeeb5ba34c2e4f9516edcb8b56f22467e5c93a21d04b3b3"} Nov 02 22:30:00.571344 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.571307 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt" event={"ID":"095af893-e150-43b9-b479-263a270d05c4","Type":"ContainerDied","Data":"834c15c7e2d55eb910c78444011a9d40e3524a64271d25455e569f01b7b36b43"} Nov 02 22:30:00.571344 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.571324 4935 scope.go:122] "RemoveContainer" containerID="c713f4ce3c43a590bdeeb5ba34c2e4f9516edcb8b56f22467e5c93a21d04b3b3" Nov 02 22:30:00.573005 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.572976 4935 generic.go:356] "Generic (PLEG): container finished" podID="3920de75-9c3e-4826-8de5-22fcc6d50b0e" containerID="fb046d23c6efc411fd4cf323c6be57faf02a543f14805601fd86e5e409bbc374" exitCode=0 Nov 02 22:30:00.573124 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.573008 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" event={"ID":"3920de75-9c3e-4826-8de5-22fcc6d50b0e","Type":"ContainerDied","Data":"fb046d23c6efc411fd4cf323c6be57faf02a543f14805601fd86e5e409bbc374"} Nov 02 22:30:00.573124 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.573025 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" Nov 02 22:30:00.573124 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.573039 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm" event={"ID":"3920de75-9c3e-4826-8de5-22fcc6d50b0e","Type":"ContainerDied","Data":"00fff1e9520b13deb9a72a0024758972063edb8a39da2a2ccda0098accd25014"} Nov 02 22:30:00.583816 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.583789 4935 scope.go:122] "RemoveContainer" containerID="c713f4ce3c43a590bdeeb5ba34c2e4f9516edcb8b56f22467e5c93a21d04b3b3" Nov 02 22:30:00.584169 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:30:00.584111 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c713f4ce3c43a590bdeeb5ba34c2e4f9516edcb8b56f22467e5c93a21d04b3b3\": container with ID starting with c713f4ce3c43a590bdeeb5ba34c2e4f9516edcb8b56f22467e5c93a21d04b3b3 not found: ID does not exist" containerID="c713f4ce3c43a590bdeeb5ba34c2e4f9516edcb8b56f22467e5c93a21d04b3b3" Nov 02 22:30:00.584308 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.584286 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c713f4ce3c43a590bdeeb5ba34c2e4f9516edcb8b56f22467e5c93a21d04b3b3"} err="failed to get container status \"c713f4ce3c43a590bdeeb5ba34c2e4f9516edcb8b56f22467e5c93a21d04b3b3\": rpc error: code = NotFound desc = could not find container \"c713f4ce3c43a590bdeeb5ba34c2e4f9516edcb8b56f22467e5c93a21d04b3b3\": container with ID starting with c713f4ce3c43a590bdeeb5ba34c2e4f9516edcb8b56f22467e5c93a21d04b3b3 not found: ID does not exist" Nov 02 22:30:00.584432 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.584404 4935 scope.go:122] "RemoveContainer" containerID="fb046d23c6efc411fd4cf323c6be57faf02a543f14805601fd86e5e409bbc374" Nov 02 22:30:00.597000 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.596975 4935 scope.go:122] "RemoveContainer" containerID="fb046d23c6efc411fd4cf323c6be57faf02a543f14805601fd86e5e409bbc374" Nov 02 22:30:00.597402 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:30:00.597368 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb046d23c6efc411fd4cf323c6be57faf02a543f14805601fd86e5e409bbc374\": container with ID starting with fb046d23c6efc411fd4cf323c6be57faf02a543f14805601fd86e5e409bbc374 not found: ID does not exist" containerID="fb046d23c6efc411fd4cf323c6be57faf02a543f14805601fd86e5e409bbc374" Nov 02 22:30:00.597519 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.597411 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb046d23c6efc411fd4cf323c6be57faf02a543f14805601fd86e5e409bbc374"} err="failed to get container status \"fb046d23c6efc411fd4cf323c6be57faf02a543f14805601fd86e5e409bbc374\": rpc error: code = NotFound desc = could not find container \"fb046d23c6efc411fd4cf323c6be57faf02a543f14805601fd86e5e409bbc374\": container with ID starting with fb046d23c6efc411fd4cf323c6be57faf02a543f14805601fd86e5e409bbc374 not found: ID does not exist" Nov 02 22:30:00.688263 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.688229 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/095af893-e150-43b9-b479-263a270d05c4-kube-api-access-bwh4q\" (UniqueName: \"kubernetes.io/projected/095af893-e150-43b9-b479-263a270d05c4-kube-api-access-bwh4q\") pod \"095af893-e150-43b9-b479-263a270d05c4\" (UID: \"095af893-e150-43b9-b479-263a270d05c4\") " Nov 02 22:30:00.688443 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.688316 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3920de75-9c3e-4826-8de5-22fcc6d50b0e-home\" (UniqueName: \"kubernetes.io/empty-dir/3920de75-9c3e-4826-8de5-22fcc6d50b0e-home\") pod \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\" (UID: \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\") " Nov 02 22:30:00.688443 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.688347 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/095af893-e150-43b9-b479-263a270d05c4-tls-certs\" (UniqueName: \"kubernetes.io/secret/095af893-e150-43b9-b479-263a270d05c4-tls-certs\") pod \"095af893-e150-43b9-b479-263a270d05c4\" (UID: \"095af893-e150-43b9-b479-263a270d05c4\") " Nov 02 22:30:00.688443 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.688388 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095af893-e150-43b9-b479-263a270d05c4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/095af893-e150-43b9-b479-263a270d05c4-dshm\") pod \"095af893-e150-43b9-b479-263a270d05c4\" (UID: \"095af893-e150-43b9-b479-263a270d05c4\") " Nov 02 22:30:00.688638 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.688463 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3920de75-9c3e-4826-8de5-22fcc6d50b0e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3920de75-9c3e-4826-8de5-22fcc6d50b0e-model-cache\") pod \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\" (UID: \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\") " Nov 02 22:30:00.688638 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.688509 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095af893-e150-43b9-b479-263a270d05c4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095af893-e150-43b9-b479-263a270d05c4-tmp-dir\") pod \"095af893-e150-43b9-b479-263a270d05c4\" (UID: \"095af893-e150-43b9-b479-263a270d05c4\") " Nov 02 22:30:00.688638 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.688545 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095af893-e150-43b9-b479-263a270d05c4-home\" (UniqueName: \"kubernetes.io/empty-dir/095af893-e150-43b9-b479-263a270d05c4-home\") pod \"095af893-e150-43b9-b479-263a270d05c4\" (UID: \"095af893-e150-43b9-b479-263a270d05c4\") " Nov 02 22:30:00.688638 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.688584 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3920de75-9c3e-4826-8de5-22fcc6d50b0e-tls-certs\" (UniqueName: \"kubernetes.io/secret/3920de75-9c3e-4826-8de5-22fcc6d50b0e-tls-certs\") pod \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\" (UID: \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\") " Nov 02 22:30:00.688638 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.688617 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3920de75-9c3e-4826-8de5-22fcc6d50b0e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3920de75-9c3e-4826-8de5-22fcc6d50b0e-dshm\") pod \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\" (UID: \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\") " Nov 02 22:30:00.688919 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.688650 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3920de75-9c3e-4826-8de5-22fcc6d50b0e-kube-api-access-85rrw\" (UniqueName: \"kubernetes.io/projected/3920de75-9c3e-4826-8de5-22fcc6d50b0e-kube-api-access-85rrw\") pod \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\" (UID: \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\") " Nov 02 22:30:00.688919 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.688680 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3920de75-9c3e-4826-8de5-22fcc6d50b0e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3920de75-9c3e-4826-8de5-22fcc6d50b0e-tmp-dir\") pod \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\" (UID: \"3920de75-9c3e-4826-8de5-22fcc6d50b0e\") " Nov 02 22:30:00.688919 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.688716 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095af893-e150-43b9-b479-263a270d05c4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095af893-e150-43b9-b479-263a270d05c4-model-cache\") pod \"095af893-e150-43b9-b479-263a270d05c4\" (UID: \"095af893-e150-43b9-b479-263a270d05c4\") " Nov 02 22:30:00.688919 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.688828 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095af893-e150-43b9-b479-263a270d05c4-home" pod "095af893-e150-43b9-b479-263a270d05c4" (UID: "095af893-e150-43b9-b479-263a270d05c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:00.689153 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.689033 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095af893-e150-43b9-b479-263a270d05c4-tmp-dir" pod "095af893-e150-43b9-b479-263a270d05c4" (UID: "095af893-e150-43b9-b479-263a270d05c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:00.689221 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.689169 4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095af893-e150-43b9-b479-263a270d05c4-tmp-dir\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:00.689221 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.689188 4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095af893-e150-43b9-b479-263a270d05c4-home\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:00.689339 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.689299 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3920de75-9c3e-4826-8de5-22fcc6d50b0e-tmp-dir" pod "3920de75-9c3e-4826-8de5-22fcc6d50b0e" (UID: "3920de75-9c3e-4826-8de5-22fcc6d50b0e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:00.689488 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.689390 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3920de75-9c3e-4826-8de5-22fcc6d50b0e-model-cache" pod "3920de75-9c3e-4826-8de5-22fcc6d50b0e" (UID: "3920de75-9c3e-4826-8de5-22fcc6d50b0e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:00.689585 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.689519 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095af893-e150-43b9-b479-263a270d05c4-model-cache" pod "095af893-e150-43b9-b479-263a270d05c4" (UID: "095af893-e150-43b9-b479-263a270d05c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:00.689793 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.689764 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3920de75-9c3e-4826-8de5-22fcc6d50b0e-home" pod "3920de75-9c3e-4826-8de5-22fcc6d50b0e" (UID: "3920de75-9c3e-4826-8de5-22fcc6d50b0e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:00.692007 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.691976 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3920de75-9c3e-4826-8de5-22fcc6d50b0e-kube-api-access-85rrw" pod "3920de75-9c3e-4826-8de5-22fcc6d50b0e" (UID: "3920de75-9c3e-4826-8de5-22fcc6d50b0e"). InnerVolumeSpecName "kube-api-access-85rrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:00.692268 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.692237 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095af893-e150-43b9-b479-263a270d05c4-dshm" pod "095af893-e150-43b9-b479-263a270d05c4" (UID: "095af893-e150-43b9-b479-263a270d05c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:00.692360 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.692334 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/095af893-e150-43b9-b479-263a270d05c4-tls-certs" pod "095af893-e150-43b9-b479-263a270d05c4" (UID: "095af893-e150-43b9-b479-263a270d05c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:30:00.692665 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.692642 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3920de75-9c3e-4826-8de5-22fcc6d50b0e-dshm" pod "3920de75-9c3e-4826-8de5-22fcc6d50b0e" (UID: "3920de75-9c3e-4826-8de5-22fcc6d50b0e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:00.692746 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.692733 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3920de75-9c3e-4826-8de5-22fcc6d50b0e-tls-certs" pod "3920de75-9c3e-4826-8de5-22fcc6d50b0e" (UID: "3920de75-9c3e-4826-8de5-22fcc6d50b0e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:30:00.693000 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.692977 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095af893-e150-43b9-b479-263a270d05c4-kube-api-access-bwh4q" pod "095af893-e150-43b9-b479-263a270d05c4" (UID: "095af893-e150-43b9-b479-263a270d05c4"). InnerVolumeSpecName "kube-api-access-bwh4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:00.790290 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.790253 4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095af893-e150-43b9-b479-263a270d05c4-model-cache\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:00.790290 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.790281 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwh4q\" (UniqueName: \"kubernetes.io/projected/095af893-e150-43b9-b479-263a270d05c4-kube-api-access-bwh4q\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:00.790290 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.790292 4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3920de75-9c3e-4826-8de5-22fcc6d50b0e-home\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:00.790518 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.790300 4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095af893-e150-43b9-b479-263a270d05c4-tls-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:00.790518 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.790308 4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095af893-e150-43b9-b479-263a270d05c4-dshm\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:00.790518 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.790317 4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3920de75-9c3e-4826-8de5-22fcc6d50b0e-model-cache\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:00.790518 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.790325 4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3920de75-9c3e-4826-8de5-22fcc6d50b0e-tls-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:00.790518 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.790334 4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3920de75-9c3e-4826-8de5-22fcc6d50b0e-dshm\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:00.790518 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.790342 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85rrw\" (UniqueName: \"kubernetes.io/projected/3920de75-9c3e-4826-8de5-22fcc6d50b0e-kube-api-access-85rrw\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:00.790518 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.790350 4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3920de75-9c3e-4826-8de5-22fcc6d50b0e-tmp-dir\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:00.919939 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.919908 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k"] Nov 02 22:30:00.922099 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:30:00.922069 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd425cbe_3140_4b2c_86c4_60f5261795f4.slice/crio-ed903ae92ac0ef456f257bd64aad118e33a3a76f2bac0bc8e5d26ab2e7fcb744 WatchSource:0}: Error finding container ed903ae92ac0ef456f257bd64aad118e33a3a76f2bac0bc8e5d26ab2e7fcb744: Status 404 returned error can't find the container with id ed903ae92ac0ef456f257bd64aad118e33a3a76f2bac0bc8e5d26ab2e7fcb744 Nov 02 22:30:00.986633 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.986603 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt"] Nov 02 22:30:00.990755 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:00.990731 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/distinct-2-w4-d6cb38-kserve-6dc55444b-jqfzt"] Nov 02 22:30:01.090722 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:01.090688 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm"] Nov 02 22:30:01.096492 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:01.096456 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d6cb38/simulator-w4-d6cb38-kserve-7588b6879f-4mmqm"] Nov 02 22:30:01.582847 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:01.582805 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k" event={"ID":"cd425cbe-3140-4b2c-86c4-60f5261795f4","Type":"ContainerStarted","Data":"639a5fa4d49bbfacde939376462a7de2405fdfc1100bae90994308383b6695e8"} Nov 02 22:30:01.583293 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:01.582873 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k" event={"ID":"cd425cbe-3140-4b2c-86c4-60f5261795f4","Type":"ContainerStarted","Data":"ed903ae92ac0ef456f257bd64aad118e33a3a76f2bac0bc8e5d26ab2e7fcb744"} Nov 02 22:30:01.691164 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:01.691098 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k" podStartSLOduration=1.69108446 podStartE2EDuration="1.69108446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:30:01.689065192 +0000 UTC m=+4894.128639914" watchObservedRunningTime="2026-11-02 22:30:01.69108446 +0000 UTC m=+4894.130659195" Nov 02 22:30:01.777530 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:01.777492 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="095af893-e150-43b9-b479-263a270d05c4" path="/var/lib/kubelet/pods/095af893-e150-43b9-b479-263a270d05c4/volumes" Nov 02 22:30:01.777938 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:01.777924 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3920de75-9c3e-4826-8de5-22fcc6d50b0e" path="/var/lib/kubelet/pods/3920de75-9c3e-4826-8de5-22fcc6d50b0e/volumes" Nov 02 22:30:02.590823 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:02.590789 4935 generic.go:356] "Generic (PLEG): container finished" podID="cd425cbe-3140-4b2c-86c4-60f5261795f4" containerID="639a5fa4d49bbfacde939376462a7de2405fdfc1100bae90994308383b6695e8" exitCode=60 Nov 02 22:30:02.591230 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:02.590865 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k" event={"ID":"cd425cbe-3140-4b2c-86c4-60f5261795f4","Type":"ContainerDied","Data":"639a5fa4d49bbfacde939376462a7de2405fdfc1100bae90994308383b6695e8"} Nov 02 22:30:02.591230 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:02.591178 4935 scope.go:122] "RemoveContainer" containerID="639a5fa4d49bbfacde939376462a7de2405fdfc1100bae90994308383b6695e8" Nov 02 22:30:03.599244 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:03.599212 4935 generic.go:356] "Generic (PLEG): container finished" podID="cd425cbe-3140-4b2c-86c4-60f5261795f4" containerID="3b128ded5693ea82f4b9e3c92de5e54c487d00fe3da32dd138609803425a4d0f" exitCode=60 Nov 02 22:30:03.599708 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:03.599287 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k" event={"ID":"cd425cbe-3140-4b2c-86c4-60f5261795f4","Type":"ContainerDied","Data":"3b128ded5693ea82f4b9e3c92de5e54c487d00fe3da32dd138609803425a4d0f"} Nov 02 22:30:03.599708 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:03.599348 4935 scope.go:122] "RemoveContainer" containerID="639a5fa4d49bbfacde939376462a7de2405fdfc1100bae90994308383b6695e8" Nov 02 22:30:03.599708 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:03.599649 4935 scope.go:122] "RemoveContainer" containerID="3b128ded5693ea82f4b9e3c92de5e54c487d00fe3da32dd138609803425a4d0f" Nov 02 22:30:03.599924 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:30:03.599903 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k_odh-ai-gateway-infra(cd425cbe-3140-4b2c-86c4-60f5261795f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k" podUID="cd425cbe-3140-4b2c-86c4-60f5261795f4" Nov 02 22:30:03.908559 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:03.908477 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:04.183609 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:04.183525 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:04.607447 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:04.607414 4935 scope.go:122] "RemoveContainer" containerID="3b128ded5693ea82f4b9e3c92de5e54c487d00fe3da32dd138609803425a4d0f" Nov 02 22:30:04.607912 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:30:04.607725 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k_odh-ai-gateway-infra(cd425cbe-3140-4b2c-86c4-60f5261795f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k" podUID="cd425cbe-3140-4b2c-86c4-60f5261795f4" Nov 02 22:30:05.041286 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:05.041248 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:09.129363 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:09.129282 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:09.446657 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:09.446566 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:12.354263 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:12.354219 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:13.533438 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:13.533395 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:14.770228 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:14.770198 4935 scope.go:122] "RemoveContainer" containerID="3b128ded5693ea82f4b9e3c92de5e54c487d00fe3da32dd138609803425a4d0f" Nov 02 22:30:14.927153 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:14.927098 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:15.412875 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:15.412787 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:15.623869 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:15.623834 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:15.675250 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:15.675178 4935 generic.go:356] "Generic (PLEG): container finished" podID="cd425cbe-3140-4b2c-86c4-60f5261795f4" containerID="3af38dda17e7d73c08dd8658c13c5856390aeb8f641f303c995fee2e98f3c02d" exitCode=60 Nov 02 22:30:15.675402 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:15.675248 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k" event={"ID":"cd425cbe-3140-4b2c-86c4-60f5261795f4","Type":"ContainerDied","Data":"3af38dda17e7d73c08dd8658c13c5856390aeb8f641f303c995fee2e98f3c02d"} Nov 02 22:30:15.675402 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:15.675308 4935 scope.go:122] "RemoveContainer" containerID="3b128ded5693ea82f4b9e3c92de5e54c487d00fe3da32dd138609803425a4d0f" Nov 02 22:30:15.675679 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:15.675661 4935 scope.go:122] "RemoveContainer" containerID="3af38dda17e7d73c08dd8658c13c5856390aeb8f641f303c995fee2e98f3c02d" Nov 02 22:30:15.675989 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:30:15.675965 4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k_odh-ai-gateway-infra(cd425cbe-3140-4b2c-86c4-60f5261795f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k" podUID="cd425cbe-3140-4b2c-86c4-60f5261795f4" Nov 02 22:30:16.792810 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:16.792776 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k"] Nov 02 22:30:16.938347 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:16.938318 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k" Nov 02 22:30:17.063194 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:17.063092 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd425cbe-3140-4b2c-86c4-60f5261795f4-kube-api-access-vd6km\" (UniqueName: \"kubernetes.io/projected/cd425cbe-3140-4b2c-86c4-60f5261795f4-kube-api-access-vd6km\") pod \"cd425cbe-3140-4b2c-86c4-60f5261795f4\" (UID: \"cd425cbe-3140-4b2c-86c4-60f5261795f4\") " Nov 02 22:30:17.063194 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:17.063179 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd425cbe-3140-4b2c-86c4-60f5261795f4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd425cbe-3140-4b2c-86c4-60f5261795f4-maas-api-service-ca\") pod \"cd425cbe-3140-4b2c-86c4-60f5261795f4\" (UID: \"cd425cbe-3140-4b2c-86c4-60f5261795f4\") " Nov 02 22:30:17.063550 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:17.063526 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd425cbe-3140-4b2c-86c4-60f5261795f4-maas-api-service-ca" pod "cd425cbe-3140-4b2c-86c4-60f5261795f4" (UID: "cd425cbe-3140-4b2c-86c4-60f5261795f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:17.065274 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:17.065247 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd425cbe-3140-4b2c-86c4-60f5261795f4-kube-api-access-vd6km" pod "cd425cbe-3140-4b2c-86c4-60f5261795f4" (UID: "cd425cbe-3140-4b2c-86c4-60f5261795f4"). InnerVolumeSpecName "kube-api-access-vd6km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:17.164835 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:17.164804 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd6km\" (UniqueName: \"kubernetes.io/projected/cd425cbe-3140-4b2c-86c4-60f5261795f4-kube-api-access-vd6km\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:17.164835 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:17.164833 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd425cbe-3140-4b2c-86c4-60f5261795f4-maas-api-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:17.689793 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:17.689759 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k" event={"ID":"cd425cbe-3140-4b2c-86c4-60f5261795f4","Type":"ContainerDied","Data":"ed903ae92ac0ef456f257bd64aad118e33a3a76f2bac0bc8e5d26ab2e7fcb744"} Nov 02 22:30:17.689964 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:17.689818 4935 scope.go:122] "RemoveContainer" containerID="3af38dda17e7d73c08dd8658c13c5856390aeb8f641f303c995fee2e98f3c02d" Nov 02 22:30:17.689964 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:17.689772 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k" Nov 02 22:30:17.842254 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:17.842222 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k"] Nov 02 22:30:17.888545 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:17.888513 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f51151-29894310-qxj9k"] Nov 02 22:30:19.775271 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:19.775191 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd425cbe-3140-4b2c-86c4-60f5261795f4" path="/var/lib/kubelet/pods/cd425cbe-3140-4b2c-86c4-60f5261795f4/volumes" Nov 02 22:30:20.001533 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:20.001495 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp"] Nov 02 22:30:20.114247 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:20.114118 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:20.480667 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:20.480578 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:21.269512 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:21.269474 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:22.058312 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:22.058271 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:22.265824 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:22.265786 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:23.204301 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:23.204267 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:23.405778 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:23.405743 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:25.029788 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:25.029751 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:25.238889 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:25.238850 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:25.660662 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:25.660627 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:25.865636 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:25.865600 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:26.171921 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:26.171884 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:27.458148 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:27.458107 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:28.293720 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:28.293681 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:28.469369 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:30:28.469326 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:30:28.510177 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:28.510118 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:28.761055 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:28.760971 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:30.405346 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:30.405303 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp"] Nov 02 22:30:30.405792 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:30.405506 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp" podUID="58c2190d-dfa6-4bcd-8469-edd252023956" containerName="maas-api" containerID="cri-o://0a2168fa40f1e58d73336fc5325d3fd7991c6fa0d35fbf5dec7d35da65faabed" gracePeriod=30 Nov 02 22:30:30.583733 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:30.581478 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:30.694858 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:30.694824 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp" Nov 02 22:30:30.770544 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:30.770510 4935 generic.go:356] "Generic (PLEG): container finished" podID="58c2190d-dfa6-4bcd-8469-edd252023956" containerID="0a2168fa40f1e58d73336fc5325d3fd7991c6fa0d35fbf5dec7d35da65faabed" exitCode=0 Nov 02 22:30:30.770724 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:30.770555 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp" Nov 02 22:30:30.770724 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:30.770601 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp" event={"ID":"58c2190d-dfa6-4bcd-8469-edd252023956","Type":"ContainerDied","Data":"0a2168fa40f1e58d73336fc5325d3fd7991c6fa0d35fbf5dec7d35da65faabed"} Nov 02 22:30:30.770724 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:30.770664 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp" event={"ID":"58c2190d-dfa6-4bcd-8469-edd252023956","Type":"ContainerDied","Data":"ebf944d7619f4187180f3b428cf00f41fb78ecf440c312ba70450905dd7b3765"} Nov 02 22:30:30.770724 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:30.770698 4935 scope.go:122] "RemoveContainer" containerID="0a2168fa40f1e58d73336fc5325d3fd7991c6fa0d35fbf5dec7d35da65faabed" Nov 02 22:30:30.780646 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:30.780619 4935 scope.go:122] "RemoveContainer" containerID="0a2168fa40f1e58d73336fc5325d3fd7991c6fa0d35fbf5dec7d35da65faabed" Nov 02 22:30:30.780940 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:30:30.780919 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a2168fa40f1e58d73336fc5325d3fd7991c6fa0d35fbf5dec7d35da65faabed\": container with ID starting with 0a2168fa40f1e58d73336fc5325d3fd7991c6fa0d35fbf5dec7d35da65faabed not found: ID does not exist" containerID="0a2168fa40f1e58d73336fc5325d3fd7991c6fa0d35fbf5dec7d35da65faabed" Nov 02 22:30:30.781008 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:30.780949 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a2168fa40f1e58d73336fc5325d3fd7991c6fa0d35fbf5dec7d35da65faabed"} err="failed to get container status \"0a2168fa40f1e58d73336fc5325d3fd7991c6fa0d35fbf5dec7d35da65faabed\": rpc error: code = NotFound desc = could not find container \"0a2168fa40f1e58d73336fc5325d3fd7991c6fa0d35fbf5dec7d35da65faabed\": container with ID starting with 0a2168fa40f1e58d73336fc5325d3fd7991c6fa0d35fbf5dec7d35da65faabed not found: ID does not exist" Nov 02 22:30:30.807948 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:30.807907 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58c2190d-dfa6-4bcd-8469-edd252023956-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58c2190d-dfa6-4bcd-8469-edd252023956-maas-api-tls\") pod \"58c2190d-dfa6-4bcd-8469-edd252023956\" (UID: \"58c2190d-dfa6-4bcd-8469-edd252023956\") " Nov 02 22:30:30.808174 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:30.807980 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58c2190d-dfa6-4bcd-8469-edd252023956-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58c2190d-dfa6-4bcd-8469-edd252023956-openshift-service-ca\") pod \"58c2190d-dfa6-4bcd-8469-edd252023956\" (UID: \"58c2190d-dfa6-4bcd-8469-edd252023956\") " Nov 02 22:30:30.808174 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:30.808010 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58c2190d-dfa6-4bcd-8469-edd252023956-kube-api-access-zhfbg\" (UniqueName: \"kubernetes.io/projected/58c2190d-dfa6-4bcd-8469-edd252023956-kube-api-access-zhfbg\") pod \"58c2190d-dfa6-4bcd-8469-edd252023956\" (UID: \"58c2190d-dfa6-4bcd-8469-edd252023956\") " Nov 02 22:30:30.808174 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:30.808059 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58c2190d-dfa6-4bcd-8469-edd252023956-metrics-certs\" (UniqueName: \"kubernetes.io/secret/58c2190d-dfa6-4bcd-8469-edd252023956-metrics-certs\") pod \"58c2190d-dfa6-4bcd-8469-edd252023956\" (UID: \"58c2190d-dfa6-4bcd-8469-edd252023956\") " Nov 02 22:30:30.808389 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:30.808363 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58c2190d-dfa6-4bcd-8469-edd252023956-openshift-service-ca" pod "58c2190d-dfa6-4bcd-8469-edd252023956" (UID: "58c2190d-dfa6-4bcd-8469-edd252023956"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:30.808638 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:30.808497 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58c2190d-dfa6-4bcd-8469-edd252023956-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:30.818319 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:30.818277 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58c2190d-dfa6-4bcd-8469-edd252023956-maas-api-tls" pod "58c2190d-dfa6-4bcd-8469-edd252023956" (UID: "58c2190d-dfa6-4bcd-8469-edd252023956"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:30:30.818319 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:30.818301 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58c2190d-dfa6-4bcd-8469-edd252023956-metrics-certs" pod "58c2190d-dfa6-4bcd-8469-edd252023956" (UID: "58c2190d-dfa6-4bcd-8469-edd252023956"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:30:30.818319 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:30.818305 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c2190d-dfa6-4bcd-8469-edd252023956-kube-api-access-zhfbg" pod "58c2190d-dfa6-4bcd-8469-edd252023956" (UID: "58c2190d-dfa6-4bcd-8469-edd252023956"). InnerVolumeSpecName "kube-api-access-zhfbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:30.910049 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:30.909972 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58c2190d-dfa6-4bcd-8469-edd252023956-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:30.910049 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:30.910007 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhfbg\" (UniqueName: \"kubernetes.io/projected/58c2190d-dfa6-4bcd-8469-edd252023956-kube-api-access-zhfbg\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:30.910049 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:30.910024 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58c2190d-dfa6-4bcd-8469-edd252023956-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:31.183431 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:31.183353 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp"] Nov 02 22:30:31.188434 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:31.188407 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6e535d-5bf4784df8-p5dwp"] Nov 02 22:30:31.776977 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:31.776942 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58c2190d-dfa6-4bcd-8469-edd252023956" path="/var/lib/kubelet/pods/58c2190d-dfa6-4bcd-8469-edd252023956/volumes" Nov 02 22:30:32.432993 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:32.432957 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:32.903240 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:32.903201 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:33.117417 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:33.117380 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:34.506253 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:34.506214 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:34.976091 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:34.976059 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:35.022303 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.022267 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" podUID="244817d6-8757-4aab-bfed-038ee1a69bb9" containerName="main" containerID="cri-o://34f0f47b99a3cfef076073f2dc2b441e0e4431e534443270be55f3a9f97d4462" gracePeriod=45 Nov 02 22:30:35.279110 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.279081 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" Nov 02 22:30:35.459017 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.458979 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244817d6-8757-4aab-bfed-038ee1a69bb9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/244817d6-8757-4aab-bfed-038ee1a69bb9-model-cache\") pod \"244817d6-8757-4aab-bfed-038ee1a69bb9\" (UID: \"244817d6-8757-4aab-bfed-038ee1a69bb9\") " Nov 02 22:30:35.459216 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.459061 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/244817d6-8757-4aab-bfed-038ee1a69bb9-tls-certs\" (UniqueName: \"kubernetes.io/secret/244817d6-8757-4aab-bfed-038ee1a69bb9-tls-certs\") pod \"244817d6-8757-4aab-bfed-038ee1a69bb9\" (UID: \"244817d6-8757-4aab-bfed-038ee1a69bb9\") " Nov 02 22:30:35.459216 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.459109 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244817d6-8757-4aab-bfed-038ee1a69bb9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/244817d6-8757-4aab-bfed-038ee1a69bb9-dshm\") pod \"244817d6-8757-4aab-bfed-038ee1a69bb9\" (UID: \"244817d6-8757-4aab-bfed-038ee1a69bb9\") " Nov 02 22:30:35.459344 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.459321 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244817d6-8757-4aab-bfed-038ee1a69bb9-model-cache" pod "244817d6-8757-4aab-bfed-038ee1a69bb9" (UID: "244817d6-8757-4aab-bfed-038ee1a69bb9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:35.459401 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.459324 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244817d6-8757-4aab-bfed-038ee1a69bb9-home\" (UniqueName: \"kubernetes.io/empty-dir/244817d6-8757-4aab-bfed-038ee1a69bb9-home\") pod \"244817d6-8757-4aab-bfed-038ee1a69bb9\" (UID: \"244817d6-8757-4aab-bfed-038ee1a69bb9\") " Nov 02 22:30:35.459441 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.459399 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244817d6-8757-4aab-bfed-038ee1a69bb9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/244817d6-8757-4aab-bfed-038ee1a69bb9-tmp-dir\") pod \"244817d6-8757-4aab-bfed-038ee1a69bb9\" (UID: \"244817d6-8757-4aab-bfed-038ee1a69bb9\") " Nov 02 22:30:35.459492 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.459448 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244817d6-8757-4aab-bfed-038ee1a69bb9-kube-api-access-fcbhk\" (UniqueName: \"kubernetes.io/projected/244817d6-8757-4aab-bfed-038ee1a69bb9-kube-api-access-fcbhk\") pod \"244817d6-8757-4aab-bfed-038ee1a69bb9\" (UID: \"244817d6-8757-4aab-bfed-038ee1a69bb9\") " Nov 02 22:30:35.459550 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.459532 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244817d6-8757-4aab-bfed-038ee1a69bb9-home" pod "244817d6-8757-4aab-bfed-038ee1a69bb9" (UID: "244817d6-8757-4aab-bfed-038ee1a69bb9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:35.459685 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.459655 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244817d6-8757-4aab-bfed-038ee1a69bb9-tmp-dir" pod "244817d6-8757-4aab-bfed-038ee1a69bb9" (UID: "244817d6-8757-4aab-bfed-038ee1a69bb9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:35.459850 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.459836 4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/244817d6-8757-4aab-bfed-038ee1a69bb9-model-cache\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:35.459923 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.459851 4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/244817d6-8757-4aab-bfed-038ee1a69bb9-home\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:35.459923 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.459861 4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/244817d6-8757-4aab-bfed-038ee1a69bb9-tmp-dir\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:35.461450 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.461425 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/244817d6-8757-4aab-bfed-038ee1a69bb9-tls-certs" pod "244817d6-8757-4aab-bfed-038ee1a69bb9" (UID: "244817d6-8757-4aab-bfed-038ee1a69bb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:30:35.461537 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.461472 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244817d6-8757-4aab-bfed-038ee1a69bb9-dshm" pod "244817d6-8757-4aab-bfed-038ee1a69bb9" (UID: "244817d6-8757-4aab-bfed-038ee1a69bb9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:35.461537 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.461505 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244817d6-8757-4aab-bfed-038ee1a69bb9-kube-api-access-fcbhk" pod "244817d6-8757-4aab-bfed-038ee1a69bb9" (UID: "244817d6-8757-4aab-bfed-038ee1a69bb9"). InnerVolumeSpecName "kube-api-access-fcbhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:35.560668 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.560643 4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/244817d6-8757-4aab-bfed-038ee1a69bb9-tls-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:35.560668 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.560667 4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/244817d6-8757-4aab-bfed-038ee1a69bb9-dshm\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:35.561185 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.560677 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcbhk\" (UniqueName: \"kubernetes.io/projected/244817d6-8757-4aab-bfed-038ee1a69bb9-kube-api-access-fcbhk\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:35.711398 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.711357 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:35.803006 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.802973 4935 generic.go:356] "Generic (PLEG): container finished" podID="244817d6-8757-4aab-bfed-038ee1a69bb9" containerID="34f0f47b99a3cfef076073f2dc2b441e0e4431e534443270be55f3a9f97d4462" exitCode=0 Nov 02 22:30:35.803168 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.803018 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" Nov 02 22:30:35.803168 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.803058 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" event={"ID":"244817d6-8757-4aab-bfed-038ee1a69bb9","Type":"ContainerDied","Data":"34f0f47b99a3cfef076073f2dc2b441e0e4431e534443270be55f3a9f97d4462"} Nov 02 22:30:35.803168 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.803106 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" event={"ID":"244817d6-8757-4aab-bfed-038ee1a69bb9","Type":"ContainerDied","Data":"f510d045026652fc4064c5a63beda6761ca28fdcc50fffdf5e96f7f471db55bd"} Nov 02 22:30:35.803168 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.803153 4935 scope.go:122] "RemoveContainer" containerID="34f0f47b99a3cfef076073f2dc2b441e0e4431e534443270be55f3a9f97d4462" Nov 02 22:30:35.812232 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.812207 4935 scope.go:122] "RemoveContainer" containerID="34f0f47b99a3cfef076073f2dc2b441e0e4431e534443270be55f3a9f97d4462" Nov 02 22:30:35.812545 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:30:35.812519 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34f0f47b99a3cfef076073f2dc2b441e0e4431e534443270be55f3a9f97d4462\": container with ID starting with 34f0f47b99a3cfef076073f2dc2b441e0e4431e534443270be55f3a9f97d4462 not found: ID does not exist" containerID="34f0f47b99a3cfef076073f2dc2b441e0e4431e534443270be55f3a9f97d4462" Nov 02 22:30:35.812628 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.812562 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34f0f47b99a3cfef076073f2dc2b441e0e4431e534443270be55f3a9f97d4462"} err="failed to get container status \"34f0f47b99a3cfef076073f2dc2b441e0e4431e534443270be55f3a9f97d4462\": rpc error: code = NotFound desc = could not find container \"34f0f47b99a3cfef076073f2dc2b441e0e4431e534443270be55f3a9f97d4462\": container with ID starting with 34f0f47b99a3cfef076073f2dc2b441e0e4431e534443270be55f3a9f97d4462 not found: ID does not exist" Nov 02 22:30:35.955437 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.955399 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp"] Nov 02 22:30:35.958269 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:35.958242 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp"] Nov 02 22:30:36.222323 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:36.222237 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-0f0874/rate-test-model-0f0874-kserve-ff6c65965-qrqzp" podUID="244817d6-8757-4aab-bfed-038ee1a69bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.149:8000/ready\": context deadline exceeded" Nov 02 22:30:37.775856 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:37.775823 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="244817d6-8757-4aab-bfed-038ee1a69bb9" path="/var/lib/kubelet/pods/244817d6-8757-4aab-bfed-038ee1a69bb9/volumes" Nov 02 22:30:37.891524 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:37.891493 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:41.575151 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:41.575098 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:41.775637 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:41.775576 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:42.220968 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:42.220935 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:42.446688 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:42.446653 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:44.489782 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:44.489745 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:44.999921 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:44.999880 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:47.663373 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:47.663335 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:49.606172 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:49.606125 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:51.983844 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:51.983799 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:52.205676 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:52.205641 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:54.620717 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:54.620675 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:58.467979 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:30:58.467941 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:30:59.726450 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:30:59.726412 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:04.724381 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:04.724342 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:07.824989 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:07.824955 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:08.018012 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:08.017976 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:10.921547 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:10.921510 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:15.929898 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:15.929856 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:18.654507 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:18.654463 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:18.873731 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:18.873694 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:20.246913 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:20.246877 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:20.787278 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:20.787240 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:21.162126 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:21.162046 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:21.365635 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:21.365603 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:26.131658 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:26.131616 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf"] Nov 02 22:31:26.178080 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:26.178048 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f"] Nov 02 22:31:26.374965 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:26.374929 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:26.654103 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:26.654070 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:28.153958 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:28.153918 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:28.469007 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:31:28.468913 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:31:31.636967 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:31.636934 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:34.035508 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:34.035457 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:35.066362 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:35.066322 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:35.275653 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:35.275618 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:36.963297 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:36.963264 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:39.564242 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:39.564150 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:41.151953 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.151895 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" podUID="7cb3ac21-f737-4fc5-927e-4a8388582812" containerName="main" containerID="cri-o://9fdb74d9d26777d29405d11d86f7f00945bcf08258515a370baae8069c37f14b" gracePeriod=45 Nov 02 22:31:41.197322 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.197269 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" podUID="aab5db02-4890-45e1-bcf8-1c43c335a393" containerName="main" containerID="cri-o://d5c8f4499b8bfe4c3107ad9bda7019e0afd3bf3d1a4f0b2e65d80c138de16804" gracePeriod=45 Nov 02 22:31:41.479291 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.479267 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" Nov 02 22:31:41.482288 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.482267 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" Nov 02 22:31:41.628314 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.628284 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cb3ac21-f737-4fc5-927e-4a8388582812-kube-api-access-td6xr\" (UniqueName: \"kubernetes.io/projected/7cb3ac21-f737-4fc5-927e-4a8388582812-kube-api-access-td6xr\") pod \"7cb3ac21-f737-4fc5-927e-4a8388582812\" (UID: \"7cb3ac21-f737-4fc5-927e-4a8388582812\") " Nov 02 22:31:41.628503 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.628334 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab5db02-4890-45e1-bcf8-1c43c335a393-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aab5db02-4890-45e1-bcf8-1c43c335a393-dshm\") pod \"aab5db02-4890-45e1-bcf8-1c43c335a393\" (UID: \"aab5db02-4890-45e1-bcf8-1c43c335a393\") " Nov 02 22:31:41.628503 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.628359 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab5db02-4890-45e1-bcf8-1c43c335a393-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aab5db02-4890-45e1-bcf8-1c43c335a393-model-cache\") pod \"aab5db02-4890-45e1-bcf8-1c43c335a393\" (UID: \"aab5db02-4890-45e1-bcf8-1c43c335a393\") " Nov 02 22:31:41.628503 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.628380 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb3ac21-f737-4fc5-927e-4a8388582812-home\" (UniqueName: \"kubernetes.io/empty-dir/7cb3ac21-f737-4fc5-927e-4a8388582812-home\") pod \"7cb3ac21-f737-4fc5-927e-4a8388582812\" (UID: \"7cb3ac21-f737-4fc5-927e-4a8388582812\") " Nov 02 22:31:41.628503 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.628413 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb3ac21-f737-4fc5-927e-4a8388582812-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cb3ac21-f737-4fc5-927e-4a8388582812-tmp-dir\") pod \"7cb3ac21-f737-4fc5-927e-4a8388582812\" (UID: \"7cb3ac21-f737-4fc5-927e-4a8388582812\") " Nov 02 22:31:41.628503 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.628452 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aab5db02-4890-45e1-bcf8-1c43c335a393-tls-certs\" (UniqueName: \"kubernetes.io/secret/aab5db02-4890-45e1-bcf8-1c43c335a393-tls-certs\") pod \"aab5db02-4890-45e1-bcf8-1c43c335a393\" (UID: \"aab5db02-4890-45e1-bcf8-1c43c335a393\") " Nov 02 22:31:41.628503 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.628477 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab5db02-4890-45e1-bcf8-1c43c335a393-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aab5db02-4890-45e1-bcf8-1c43c335a393-tmp-dir\") pod \"aab5db02-4890-45e1-bcf8-1c43c335a393\" (UID: \"aab5db02-4890-45e1-bcf8-1c43c335a393\") " Nov 02 22:31:41.628503 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.628500 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb3ac21-f737-4fc5-927e-4a8388582812-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cb3ac21-f737-4fc5-927e-4a8388582812-model-cache\") pod \"7cb3ac21-f737-4fc5-927e-4a8388582812\" (UID: \"7cb3ac21-f737-4fc5-927e-4a8388582812\") " Nov 02 22:31:41.628885 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.628564 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aab5db02-4890-45e1-bcf8-1c43c335a393-kube-api-access-qkbvg\" (UniqueName: \"kubernetes.io/projected/aab5db02-4890-45e1-bcf8-1c43c335a393-kube-api-access-qkbvg\") pod \"aab5db02-4890-45e1-bcf8-1c43c335a393\" (UID: \"aab5db02-4890-45e1-bcf8-1c43c335a393\") " Nov 02 22:31:41.628885 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.628599 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb3ac21-f737-4fc5-927e-4a8388582812-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cb3ac21-f737-4fc5-927e-4a8388582812-dshm\") pod \"7cb3ac21-f737-4fc5-927e-4a8388582812\" (UID: \"7cb3ac21-f737-4fc5-927e-4a8388582812\") " Nov 02 22:31:41.628885 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.628639 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cb3ac21-f737-4fc5-927e-4a8388582812-tls-certs\" (UniqueName: \"kubernetes.io/secret/7cb3ac21-f737-4fc5-927e-4a8388582812-tls-certs\") pod \"7cb3ac21-f737-4fc5-927e-4a8388582812\" (UID: \"7cb3ac21-f737-4fc5-927e-4a8388582812\") " Nov 02 22:31:41.628885 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.628671 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab5db02-4890-45e1-bcf8-1c43c335a393-home\" (UniqueName: \"kubernetes.io/empty-dir/aab5db02-4890-45e1-bcf8-1c43c335a393-home\") pod \"aab5db02-4890-45e1-bcf8-1c43c335a393\" (UID: \"aab5db02-4890-45e1-bcf8-1c43c335a393\") " Nov 02 22:31:41.630447 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.629404 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab5db02-4890-45e1-bcf8-1c43c335a393-tmp-dir" pod "aab5db02-4890-45e1-bcf8-1c43c335a393" (UID: "aab5db02-4890-45e1-bcf8-1c43c335a393"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:31:41.630447 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.629417 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab5db02-4890-45e1-bcf8-1c43c335a393-model-cache" pod "aab5db02-4890-45e1-bcf8-1c43c335a393" (UID: "aab5db02-4890-45e1-bcf8-1c43c335a393"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:31:41.630447 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.629480 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab5db02-4890-45e1-bcf8-1c43c335a393-home" pod "aab5db02-4890-45e1-bcf8-1c43c335a393" (UID: "aab5db02-4890-45e1-bcf8-1c43c335a393"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:31:41.630447 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.629868 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb3ac21-f737-4fc5-927e-4a8388582812-tmp-dir" pod "7cb3ac21-f737-4fc5-927e-4a8388582812" (UID: "7cb3ac21-f737-4fc5-927e-4a8388582812"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:31:41.630447 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.630088 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb3ac21-f737-4fc5-927e-4a8388582812-model-cache" pod "7cb3ac21-f737-4fc5-927e-4a8388582812" (UID: "7cb3ac21-f737-4fc5-927e-4a8388582812"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:31:41.631317 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.631280 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb3ac21-f737-4fc5-927e-4a8388582812-home" pod "7cb3ac21-f737-4fc5-927e-4a8388582812" (UID: "7cb3ac21-f737-4fc5-927e-4a8388582812"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:31:41.633548 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.633502 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab5db02-4890-45e1-bcf8-1c43c335a393-dshm" pod "aab5db02-4890-45e1-bcf8-1c43c335a393" (UID: "aab5db02-4890-45e1-bcf8-1c43c335a393"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:31:41.635597 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.635571 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aab5db02-4890-45e1-bcf8-1c43c335a393-tls-certs" pod "aab5db02-4890-45e1-bcf8-1c43c335a393" (UID: "aab5db02-4890-45e1-bcf8-1c43c335a393"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:31:41.636636 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.636608 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aab5db02-4890-45e1-bcf8-1c43c335a393-kube-api-access-qkbvg" pod "aab5db02-4890-45e1-bcf8-1c43c335a393" (UID: "aab5db02-4890-45e1-bcf8-1c43c335a393"). InnerVolumeSpecName "kube-api-access-qkbvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:31:41.636923 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.636900 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb3ac21-f737-4fc5-927e-4a8388582812-kube-api-access-td6xr" pod "7cb3ac21-f737-4fc5-927e-4a8388582812" (UID: "7cb3ac21-f737-4fc5-927e-4a8388582812"). InnerVolumeSpecName "kube-api-access-td6xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:31:41.637026 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.636944 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cb3ac21-f737-4fc5-927e-4a8388582812-tls-certs" pod "7cb3ac21-f737-4fc5-927e-4a8388582812" (UID: "7cb3ac21-f737-4fc5-927e-4a8388582812"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:31:41.637158 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.637105 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb3ac21-f737-4fc5-927e-4a8388582812-dshm" pod "7cb3ac21-f737-4fc5-927e-4a8388582812" (UID: "7cb3ac21-f737-4fc5-927e-4a8388582812"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:31:41.729637 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.729547 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td6xr\" (UniqueName: \"kubernetes.io/projected/7cb3ac21-f737-4fc5-927e-4a8388582812-kube-api-access-td6xr\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:41.729637 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.729585 4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aab5db02-4890-45e1-bcf8-1c43c335a393-dshm\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:41.729637 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.729596 4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aab5db02-4890-45e1-bcf8-1c43c335a393-model-cache\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:41.729637 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.729611 4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cb3ac21-f737-4fc5-927e-4a8388582812-home\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:41.729637 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.729643 4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cb3ac21-f737-4fc5-927e-4a8388582812-tmp-dir\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:41.730093 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.729651 4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aab5db02-4890-45e1-bcf8-1c43c335a393-tls-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:41.730093 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.729668 4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aab5db02-4890-45e1-bcf8-1c43c335a393-tmp-dir\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:41.730093 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.729690 4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cb3ac21-f737-4fc5-927e-4a8388582812-model-cache\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:41.730093 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.729701 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkbvg\" (UniqueName: \"kubernetes.io/projected/aab5db02-4890-45e1-bcf8-1c43c335a393-kube-api-access-qkbvg\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:41.730093 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.729711 4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cb3ac21-f737-4fc5-927e-4a8388582812-dshm\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:41.730093 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.729721 4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cb3ac21-f737-4fc5-927e-4a8388582812-tls-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:41.730093 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:41.729728 4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aab5db02-4890-45e1-bcf8-1c43c335a393-home\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:42.132347 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:42.132312 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:42.176143 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:42.176106 4935 generic.go:356] "Generic (PLEG): container finished" podID="7cb3ac21-f737-4fc5-927e-4a8388582812" containerID="9fdb74d9d26777d29405d11d86f7f00945bcf08258515a370baae8069c37f14b" exitCode=0 Nov 02 22:31:42.176494 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:42.176169 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" Nov 02 22:31:42.176494 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:42.176187 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" event={"ID":"7cb3ac21-f737-4fc5-927e-4a8388582812","Type":"ContainerDied","Data":"9fdb74d9d26777d29405d11d86f7f00945bcf08258515a370baae8069c37f14b"} Nov 02 22:31:42.176494 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:42.176229 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf" event={"ID":"7cb3ac21-f737-4fc5-927e-4a8388582812","Type":"ContainerDied","Data":"5af46704022743e30370b8a04a98a56293d8d6af2826135e6951dbc918b354a7"} Nov 02 22:31:42.176494 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:42.176248 4935 scope.go:122] "RemoveContainer" containerID="9fdb74d9d26777d29405d11d86f7f00945bcf08258515a370baae8069c37f14b" Nov 02 22:31:42.177785 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:42.177739 4935 generic.go:356] "Generic (PLEG): container finished" podID="aab5db02-4890-45e1-bcf8-1c43c335a393" containerID="d5c8f4499b8bfe4c3107ad9bda7019e0afd3bf3d1a4f0b2e65d80c138de16804" exitCode=0 Nov 02 22:31:42.177874 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:42.177782 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" Nov 02 22:31:42.177874 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:42.177830 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" event={"ID":"aab5db02-4890-45e1-bcf8-1c43c335a393","Type":"ContainerDied","Data":"d5c8f4499b8bfe4c3107ad9bda7019e0afd3bf3d1a4f0b2e65d80c138de16804"} Nov 02 22:31:42.177874 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:42.177864 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f" event={"ID":"aab5db02-4890-45e1-bcf8-1c43c335a393","Type":"ContainerDied","Data":"46bedece67c5c6cd1f61abed6a62e7142fca61602118adbb50772bfd273144e9"} Nov 02 22:31:42.185872 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:42.185846 4935 scope.go:122] "RemoveContainer" containerID="9fdb74d9d26777d29405d11d86f7f00945bcf08258515a370baae8069c37f14b" Nov 02 22:31:42.186114 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:31:42.186095 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fdb74d9d26777d29405d11d86f7f00945bcf08258515a370baae8069c37f14b\": container with ID starting with 9fdb74d9d26777d29405d11d86f7f00945bcf08258515a370baae8069c37f14b not found: ID does not exist" containerID="9fdb74d9d26777d29405d11d86f7f00945bcf08258515a370baae8069c37f14b" Nov 02 22:31:42.186202 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:42.186121 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fdb74d9d26777d29405d11d86f7f00945bcf08258515a370baae8069c37f14b"} err="failed to get container status \"9fdb74d9d26777d29405d11d86f7f00945bcf08258515a370baae8069c37f14b\": rpc error: code = NotFound desc = could not find container \"9fdb74d9d26777d29405d11d86f7f00945bcf08258515a370baae8069c37f14b\": container with ID starting with 9fdb74d9d26777d29405d11d86f7f00945bcf08258515a370baae8069c37f14b not found: ID does not exist" Nov 02 22:31:42.186202 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:42.186165 4935 scope.go:122] "RemoveContainer" containerID="d5c8f4499b8bfe4c3107ad9bda7019e0afd3bf3d1a4f0b2e65d80c138de16804" Nov 02 22:31:42.194067 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:42.194049 4935 scope.go:122] "RemoveContainer" containerID="d5c8f4499b8bfe4c3107ad9bda7019e0afd3bf3d1a4f0b2e65d80c138de16804" Nov 02 22:31:42.194347 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:31:42.194321 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5c8f4499b8bfe4c3107ad9bda7019e0afd3bf3d1a4f0b2e65d80c138de16804\": container with ID starting with d5c8f4499b8bfe4c3107ad9bda7019e0afd3bf3d1a4f0b2e65d80c138de16804 not found: ID does not exist" containerID="d5c8f4499b8bfe4c3107ad9bda7019e0afd3bf3d1a4f0b2e65d80c138de16804" Nov 02 22:31:42.194410 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:42.194352 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c8f4499b8bfe4c3107ad9bda7019e0afd3bf3d1a4f0b2e65d80c138de16804"} err="failed to get container status \"d5c8f4499b8bfe4c3107ad9bda7019e0afd3bf3d1a4f0b2e65d80c138de16804\": rpc error: code = NotFound desc = could not find container \"d5c8f4499b8bfe4c3107ad9bda7019e0afd3bf3d1a4f0b2e65d80c138de16804\": container with ID starting with d5c8f4499b8bfe4c3107ad9bda7019e0afd3bf3d1a4f0b2e65d80c138de16804 not found: ID does not exist" Nov 02 22:31:42.331363 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:42.331323 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf"] Nov 02 22:31:42.333890 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:42.333863 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-0c9014/premium-w0-0c9014-kserve-75d9c897cc-7ttjf"] Nov 02 22:31:42.434772 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:42.434701 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f"] Nov 02 22:31:42.437000 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:42.436971 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-0c9014/simulator-w0-0c9014-kserve-6f66c899c6-59v7f"] Nov 02 22:31:43.777900 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:43.777869 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cb3ac21-f737-4fc5-927e-4a8388582812" path="/var/lib/kubelet/pods/7cb3ac21-f737-4fc5-927e-4a8388582812/volumes" Nov 02 22:31:43.778318 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:43.778290 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aab5db02-4890-45e1-bcf8-1c43c335a393" path="/var/lib/kubelet/pods/aab5db02-4890-45e1-bcf8-1c43c335a393/volumes" Nov 02 22:31:45.266262 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:45.266223 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:45.464268 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:45.464230 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:47.240653 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:47.240613 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:52.220086 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:52.220050 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:56.594481 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:56.594440 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:58.409769 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:58.409728 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:58.471114 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:31:58.471069 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:31:59.909219 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:31:59.909175 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:00.119524 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:32:00.119485 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:03.230803 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:32:03.230742 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:03.435876 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:32:03.435828 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:03.645429 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:32:03.645394 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:03.908565 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:32:03.908476 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:08.006538 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:32:08.006499 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:08.475882 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:32:08.475851 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:10.778511 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:32:10.778480 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:10.983784 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:32:10.983736 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:14.226108 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:32:14.226069 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:19.224025 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:32:19.223984 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:24.244029 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:32:24.243987 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:28.470362 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:32:28.470307 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:32:29.336434 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:32:29.336371 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:34.416108 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:32:34.416068 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:39.423489 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:32:39.423446 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:44.409833 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:32:44.409792 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:45.656073 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:32:45.656027 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:49.511278 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:32:49.511243 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:54.550681 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:32:54.550648 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:58.469093 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:32:58.469049 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:32:59.519296 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:32:59.519263 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:59.873543 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:32:59.873443 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:00.075207 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:00.075165 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:01.936200 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:01.936165 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:02.127817 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:02.127766 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:04.524525 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:04.524486 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:06.806981 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:06.806943 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:09.610195 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:09.610085 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:14.616440 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:14.616373 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:19.637959 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:19.637917 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:24.634085 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:24.634043 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:26.275157 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:26.273382 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:26.710704 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:26.710611 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:28.470431 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:33:28.470389 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:33:28.681980 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:28.681953 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 22:33:28.691275 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:28.691252 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 22:33:28.721851 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:28.721781 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 22:33:28.730525 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:28.730500 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 22:33:29.726144 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:29.726091 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:32.544458 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:32.544408 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:33.501032 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:33.500996 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:33.751258 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:33.751161 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:35.057176 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:35.057111 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:38.148292 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:38.148243 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:39.332013 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:39.331975 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644"] Nov 02 22:33:39.332487 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:39.332246 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644" podUID="ee1633c8-eaae-419f-8912-e506e14bd98d" containerName="maas-api" containerID="cri-o://3b5fc5a904bbb95a6ec2957153e1c061b66d78113342208eeb6dbc796b663dbe" gracePeriod=30 Nov 02 22:33:39.612911 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:39.612883 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644" Nov 02 22:33:39.713738 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:39.713700 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee1633c8-eaae-419f-8912-e506e14bd98d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee1633c8-eaae-419f-8912-e506e14bd98d-maas-api-tls\") pod \"ee1633c8-eaae-419f-8912-e506e14bd98d\" (UID: \"ee1633c8-eaae-419f-8912-e506e14bd98d\") " Nov 02 22:33:39.713955 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:39.713794 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee1633c8-eaae-419f-8912-e506e14bd98d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee1633c8-eaae-419f-8912-e506e14bd98d-metrics-certs\") pod \"ee1633c8-eaae-419f-8912-e506e14bd98d\" (UID: \"ee1633c8-eaae-419f-8912-e506e14bd98d\") " Nov 02 22:33:39.713955 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:39.713877 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee1633c8-eaae-419f-8912-e506e14bd98d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee1633c8-eaae-419f-8912-e506e14bd98d-openshift-service-ca\") pod \"ee1633c8-eaae-419f-8912-e506e14bd98d\" (UID: \"ee1633c8-eaae-419f-8912-e506e14bd98d\") " Nov 02 22:33:39.713955 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:39.713950 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee1633c8-eaae-419f-8912-e506e14bd98d-kube-api-access-hqrh4\" (UniqueName: \"kubernetes.io/projected/ee1633c8-eaae-419f-8912-e506e14bd98d-kube-api-access-hqrh4\") pod \"ee1633c8-eaae-419f-8912-e506e14bd98d\" (UID: \"ee1633c8-eaae-419f-8912-e506e14bd98d\") " Nov 02 22:33:39.714222 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:39.714187 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee1633c8-eaae-419f-8912-e506e14bd98d-openshift-service-ca" pod "ee1633c8-eaae-419f-8912-e506e14bd98d" (UID: "ee1633c8-eaae-419f-8912-e506e14bd98d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:39.714336 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:39.714322 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee1633c8-eaae-419f-8912-e506e14bd98d-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:39.716008 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:39.715979 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee1633c8-eaae-419f-8912-e506e14bd98d-maas-api-tls" pod "ee1633c8-eaae-419f-8912-e506e14bd98d" (UID: "ee1633c8-eaae-419f-8912-e506e14bd98d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:33:39.716107 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:39.716039 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee1633c8-eaae-419f-8912-e506e14bd98d-metrics-certs" pod "ee1633c8-eaae-419f-8912-e506e14bd98d" (UID: "ee1633c8-eaae-419f-8912-e506e14bd98d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:33:39.716107 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:39.716072 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee1633c8-eaae-419f-8912-e506e14bd98d-kube-api-access-hqrh4" pod "ee1633c8-eaae-419f-8912-e506e14bd98d" (UID: "ee1633c8-eaae-419f-8912-e506e14bd98d"). InnerVolumeSpecName "kube-api-access-hqrh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:39.814872 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:39.814841 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqrh4\" (UniqueName: \"kubernetes.io/projected/ee1633c8-eaae-419f-8912-e506e14bd98d-kube-api-access-hqrh4\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:39.814872 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:39.814875 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee1633c8-eaae-419f-8912-e506e14bd98d-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:39.815049 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:39.814891 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee1633c8-eaae-419f-8912-e506e14bd98d-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:39.860110 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:39.860005 4935 generic.go:356] "Generic (PLEG): container finished" podID="ee1633c8-eaae-419f-8912-e506e14bd98d" containerID="3b5fc5a904bbb95a6ec2957153e1c061b66d78113342208eeb6dbc796b663dbe" exitCode=0 Nov 02 22:33:39.860110 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:39.860077 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644" Nov 02 22:33:39.860346 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:39.860105 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644" event={"ID":"ee1633c8-eaae-419f-8912-e506e14bd98d","Type":"ContainerDied","Data":"3b5fc5a904bbb95a6ec2957153e1c061b66d78113342208eeb6dbc796b663dbe"} Nov 02 22:33:39.860346 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:39.860178 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644" event={"ID":"ee1633c8-eaae-419f-8912-e506e14bd98d","Type":"ContainerDied","Data":"02fa30c6f7720ae1d6b29aaea6cf4c032af08fb11dc294f9cd557a8f2395137d"} Nov 02 22:33:39.860346 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:39.860205 4935 scope.go:122] "RemoveContainer" containerID="3b5fc5a904bbb95a6ec2957153e1c061b66d78113342208eeb6dbc796b663dbe" Nov 02 22:33:39.872236 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:39.872213 4935 scope.go:122] "RemoveContainer" containerID="3b5fc5a904bbb95a6ec2957153e1c061b66d78113342208eeb6dbc796b663dbe" Nov 02 22:33:39.872570 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:33:39.872547 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b5fc5a904bbb95a6ec2957153e1c061b66d78113342208eeb6dbc796b663dbe\": container with ID starting with 3b5fc5a904bbb95a6ec2957153e1c061b66d78113342208eeb6dbc796b663dbe not found: ID does not exist" containerID="3b5fc5a904bbb95a6ec2957153e1c061b66d78113342208eeb6dbc796b663dbe" Nov 02 22:33:39.872710 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:39.872574 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b5fc5a904bbb95a6ec2957153e1c061b66d78113342208eeb6dbc796b663dbe"} err="failed to get container status \"3b5fc5a904bbb95a6ec2957153e1c061b66d78113342208eeb6dbc796b663dbe\": rpc error: code = NotFound desc = could not find container \"3b5fc5a904bbb95a6ec2957153e1c061b66d78113342208eeb6dbc796b663dbe\": container with ID starting with 3b5fc5a904bbb95a6ec2957153e1c061b66d78113342208eeb6dbc796b663dbe not found: ID does not exist" Nov 02 22:33:39.909153 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:39.909100 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:40.011968 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:40.011935 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644"] Nov 02 22:33:40.015604 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:40.015575 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-97995b-78dbd499d8-lv644"] Nov 02 22:33:41.315525 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:41.315488 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:41.776781 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:41.776744 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee1633c8-eaae-419f-8912-e506e14bd98d" path="/var/lib/kubelet/pods/ee1633c8-eaae-419f-8912-e506e14bd98d/volumes" Nov 02 22:33:43.841355 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:43.841316 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:44.048784 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:44.048734 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:45.378330 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:45.378286 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:45.868964 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:45.868931 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp"] Nov 02 22:33:45.869220 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:45.869178 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" podUID="2d7d3443-7c43-4244-beda-b09241df09cd" containerName="istio-proxy" containerID="cri-o://a7c1e97b68de8bf2fb9ea25af29723853435b4fadf8733be7467298737b6252f" gracePeriod=30 Nov 02 22:33:45.943005 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:45.942970 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:46.149175 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:46.149065 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:46.704310 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:46.704277 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:46.911643 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:46.911606 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:51.128145 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.128108 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 22:33:51.226024 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.225941 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-istio-envoy\") pod \"2d7d3443-7c43-4244-beda-b09241df09cd\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " Nov 02 22:33:51.226024 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.225994 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-workload-certs\") pod \"2d7d3443-7c43-4244-beda-b09241df09cd\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " Nov 02 22:33:51.226024 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.226016 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d7d3443-7c43-4244-beda-b09241df09cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d7d3443-7c43-4244-beda-b09241df09cd-istio-podinfo\") pod \"2d7d3443-7c43-4244-beda-b09241df09cd\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " Nov 02 22:33:51.226330 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.226049 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-workload-socket\") pod \"2d7d3443-7c43-4244-beda-b09241df09cd\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " Nov 02 22:33:51.226330 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.226076 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d7d3443-7c43-4244-beda-b09241df09cd-kube-api-access-5vsvr\" (UniqueName: \"kubernetes.io/projected/2d7d3443-7c43-4244-beda-b09241df09cd-kube-api-access-5vsvr\") pod \"2d7d3443-7c43-4244-beda-b09241df09cd\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " Nov 02 22:33:51.226330 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.226103 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-credential-socket\") pod \"2d7d3443-7c43-4244-beda-b09241df09cd\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " Nov 02 22:33:51.226330 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.226159 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-istio-data\") pod \"2d7d3443-7c43-4244-beda-b09241df09cd\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " Nov 02 22:33:51.226330 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.226198 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d7d3443-7c43-4244-beda-b09241df09cd-istio-token\" (UniqueName: \"kubernetes.io/projected/2d7d3443-7c43-4244-beda-b09241df09cd-istio-token\") pod \"2d7d3443-7c43-4244-beda-b09241df09cd\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " Nov 02 22:33:51.226330 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.226293 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d7d3443-7c43-4244-beda-b09241df09cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d7d3443-7c43-4244-beda-b09241df09cd-istiod-ca-cert\") pod \"2d7d3443-7c43-4244-beda-b09241df09cd\" (UID: \"2d7d3443-7c43-4244-beda-b09241df09cd\") " Nov 02 22:33:51.226658 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.226317 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-workload-certs" pod "2d7d3443-7c43-4244-beda-b09241df09cd" (UID: "2d7d3443-7c43-4244-beda-b09241df09cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:51.226658 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.226499 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-istio-data" pod "2d7d3443-7c43-4244-beda-b09241df09cd" (UID: "2d7d3443-7c43-4244-beda-b09241df09cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:51.226658 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.226525 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-credential-socket" pod "2d7d3443-7c43-4244-beda-b09241df09cd" (UID: "2d7d3443-7c43-4244-beda-b09241df09cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:51.226658 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.226571 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-workload-socket" pod "2d7d3443-7c43-4244-beda-b09241df09cd" (UID: "2d7d3443-7c43-4244-beda-b09241df09cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:51.226658 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.226637 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d7d3443-7c43-4244-beda-b09241df09cd-istiod-ca-cert" pod "2d7d3443-7c43-4244-beda-b09241df09cd" (UID: "2d7d3443-7c43-4244-beda-b09241df09cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:51.226932 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.226770 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d7d3443-7c43-4244-beda-b09241df09cd-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:51.226932 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.226793 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:51.226932 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.226808 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:51.226932 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.226823 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:51.226932 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.226835 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:51.228564 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.228542 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7d3443-7c43-4244-beda-b09241df09cd-kube-api-access-5vsvr" pod "2d7d3443-7c43-4244-beda-b09241df09cd" (UID: "2d7d3443-7c43-4244-beda-b09241df09cd"). InnerVolumeSpecName "kube-api-access-5vsvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:51.228676 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.228601 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-istio-envoy" pod "2d7d3443-7c43-4244-beda-b09241df09cd" (UID: "2d7d3443-7c43-4244-beda-b09241df09cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:51.228810 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.228766 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d7d3443-7c43-4244-beda-b09241df09cd-istio-podinfo" pod "2d7d3443-7c43-4244-beda-b09241df09cd" (UID: "2d7d3443-7c43-4244-beda-b09241df09cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:33:51.228810 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.228803 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7d3443-7c43-4244-beda-b09241df09cd-istio-token" pod "2d7d3443-7c43-4244-beda-b09241df09cd" (UID: "2d7d3443-7c43-4244-beda-b09241df09cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:51.327494 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.327459 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d7d3443-7c43-4244-beda-b09241df09cd-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:51.327494 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.327488 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d7d3443-7c43-4244-beda-b09241df09cd-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:51.327494 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.327498 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d7d3443-7c43-4244-beda-b09241df09cd-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:51.327726 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.327510 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vsvr\" (UniqueName: \"kubernetes.io/projected/2d7d3443-7c43-4244-beda-b09241df09cd-kube-api-access-5vsvr\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:51.929777 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.929733 4935 generic.go:356] "Generic (PLEG): container finished" podID="2d7d3443-7c43-4244-beda-b09241df09cd" containerID="a7c1e97b68de8bf2fb9ea25af29723853435b4fadf8733be7467298737b6252f" exitCode=0 Nov 02 22:33:51.929993 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.929786 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" Nov 02 22:33:51.929993 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.929778 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" event={"ID":"2d7d3443-7c43-4244-beda-b09241df09cd","Type":"ContainerDied","Data":"a7c1e97b68de8bf2fb9ea25af29723853435b4fadf8733be7467298737b6252f"} Nov 02 22:33:51.929993 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.929927 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp" event={"ID":"2d7d3443-7c43-4244-beda-b09241df09cd","Type":"ContainerDied","Data":"6a4612cf4269507a5825c6dc39488cea90dfd8924cc367b0f84ab76012340515"} Nov 02 22:33:51.929993 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.929959 4935 scope.go:122] "RemoveContainer" containerID="a7c1e97b68de8bf2fb9ea25af29723853435b4fadf8733be7467298737b6252f" Nov 02 22:33:51.941254 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.941222 4935 scope.go:122] "RemoveContainer" containerID="da7298f3e6a8dd1e559526f3c0842554316628379f11cdf8a37951ec0e9d91d4" Nov 02 22:33:51.950213 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.950195 4935 scope.go:122] "RemoveContainer" containerID="a7c1e97b68de8bf2fb9ea25af29723853435b4fadf8733be7467298737b6252f" Nov 02 22:33:51.950510 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:33:51.950482 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7c1e97b68de8bf2fb9ea25af29723853435b4fadf8733be7467298737b6252f\": container with ID starting with a7c1e97b68de8bf2fb9ea25af29723853435b4fadf8733be7467298737b6252f not found: ID does not exist" containerID="a7c1e97b68de8bf2fb9ea25af29723853435b4fadf8733be7467298737b6252f" Nov 02 22:33:51.950580 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.950510 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7c1e97b68de8bf2fb9ea25af29723853435b4fadf8733be7467298737b6252f"} err="failed to get container status \"a7c1e97b68de8bf2fb9ea25af29723853435b4fadf8733be7467298737b6252f\": rpc error: code = NotFound desc = could not find container \"a7c1e97b68de8bf2fb9ea25af29723853435b4fadf8733be7467298737b6252f\": container with ID starting with a7c1e97b68de8bf2fb9ea25af29723853435b4fadf8733be7467298737b6252f not found: ID does not exist" Nov 02 22:33:51.950580 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.950531 4935 scope.go:122] "RemoveContainer" containerID="da7298f3e6a8dd1e559526f3c0842554316628379f11cdf8a37951ec0e9d91d4" Nov 02 22:33:51.950828 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:33:51.950808 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da7298f3e6a8dd1e559526f3c0842554316628379f11cdf8a37951ec0e9d91d4\": container with ID starting with da7298f3e6a8dd1e559526f3c0842554316628379f11cdf8a37951ec0e9d91d4 not found: ID does not exist" containerID="da7298f3e6a8dd1e559526f3c0842554316628379f11cdf8a37951ec0e9d91d4" Nov 02 22:33:51.950893 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:51.950839 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da7298f3e6a8dd1e559526f3c0842554316628379f11cdf8a37951ec0e9d91d4"} err="failed to get container status \"da7298f3e6a8dd1e559526f3c0842554316628379f11cdf8a37951ec0e9d91d4\": rpc error: code = NotFound desc = could not find container \"da7298f3e6a8dd1e559526f3c0842554316628379f11cdf8a37951ec0e9d91d4\": container with ID starting with da7298f3e6a8dd1e559526f3c0842554316628379f11cdf8a37951ec0e9d91d4 not found: ID does not exist" Nov 02 22:33:52.016585 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:52.016555 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:52.129988 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:52.129893 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp"] Nov 02 22:33:52.175040 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:52.175005 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-97995b-openshift-default-66cfb6df99-p94sp"] Nov 02 22:33:52.705037 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:52.704990 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:53.581696 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:53.581661 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:53.774929 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:53.774892 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d7d3443-7c43-4244-beda-b09241df09cd" path="/var/lib/kubelet/pods/2d7d3443-7c43-4244-beda-b09241df09cd/volumes" Nov 02 22:33:53.824225 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:53.824196 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:57.425590 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:57.425553 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:58.202329 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:33:58.202294 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:58.470045 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:33:58.469958 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:34:01.963499 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:01.963442 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:03.628696 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:03.628661 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:03.839516 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:03.839474 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:04.095040 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:04.095003 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:05.451687 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:05.451648 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:05.960989 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:05.960956 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:06.420537 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:06.420502 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:09.084384 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:09.084349 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:11.299855 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:11.299818 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx"] Nov 02 22:34:11.346403 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:11.346376 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l"] Nov 02 22:34:11.348805 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:11.348778 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s"] Nov 02 22:34:11.643854 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:11.643773 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:13.346459 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:13.346394 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:14.230680 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:14.230619 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:19.336365 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:19.336327 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:19.563668 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:19.563628 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:20.692489 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:20.692460 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:20.889790 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:20.889757 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:24.806770 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:24.806711 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:26.320345 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.320276 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" podUID="4534e64e-1398-44a8-acb0-22db48ebea02" containerName="main" containerID="cri-o://7cd7c96e3586a3fddbfeeee1c6cb10a14875ca6e47ad4625dd90e65329b64ed4" gracePeriod=45 Nov 02 22:34:26.367269 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.367231 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" podUID="91602668-a9dd-48f5-87cb-7b7ce484db3b" containerName="main" containerID="cri-o://a488b0828abbb73269037cc2aec0ea7690e9339e6a1610f345e7371d15afcd6f" gracePeriod=45 Nov 02 22:34:26.369491 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.369460 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" podUID="bcf6e677-5fc9-447b-821c-f8ba0ecd65f8" containerName="main" containerID="cri-o://292db166abf313cb5281582a076f2403da95de31756a916f594c51b2cb90b16d" gracePeriod=45 Nov 02 22:34:26.628354 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.628321 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" Nov 02 22:34:26.659935 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.659907 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" Nov 02 22:34:26.663310 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.663289 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" Nov 02 22:34:26.676421 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.676400 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-tmp-dir\") pod \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\" (UID: \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\") " Nov 02 22:34:26.676526 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.676435 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91602668-a9dd-48f5-87cb-7b7ce484db3b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91602668-a9dd-48f5-87cb-7b7ce484db3b-model-cache\") pod \"91602668-a9dd-48f5-87cb-7b7ce484db3b\" (UID: \"91602668-a9dd-48f5-87cb-7b7ce484db3b\") " Nov 02 22:34:26.676526 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.676483 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91602668-a9dd-48f5-87cb-7b7ce484db3b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91602668-a9dd-48f5-87cb-7b7ce484db3b-tmp-dir\") pod \"91602668-a9dd-48f5-87cb-7b7ce484db3b\" (UID: \"91602668-a9dd-48f5-87cb-7b7ce484db3b\") " Nov 02 22:34:26.676526 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.676511 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-home\" (UniqueName: \"kubernetes.io/empty-dir/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-home\") pod \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\" (UID: \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\") " Nov 02 22:34:26.676694 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.676540 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91602668-a9dd-48f5-87cb-7b7ce484db3b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/91602668-a9dd-48f5-87cb-7b7ce484db3b-dshm\") pod \"91602668-a9dd-48f5-87cb-7b7ce484db3b\" (UID: \"91602668-a9dd-48f5-87cb-7b7ce484db3b\") " Nov 02 22:34:26.676694 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.676569 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4534e64e-1398-44a8-acb0-22db48ebea02-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4534e64e-1398-44a8-acb0-22db48ebea02-tmp-dir\") pod \"4534e64e-1398-44a8-acb0-22db48ebea02\" (UID: \"4534e64e-1398-44a8-acb0-22db48ebea02\") " Nov 02 22:34:26.676694 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.676614 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-kube-api-access-2p7t8\" (UniqueName: \"kubernetes.io/projected/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-kube-api-access-2p7t8\") pod \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\" (UID: \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\") " Nov 02 22:34:26.676694 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.676659 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-tls-certs\") pod \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\" (UID: \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\") " Nov 02 22:34:26.676694 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.676683 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-tmp-dir" pod "bcf6e677-5fc9-447b-821c-f8ba0ecd65f8" (UID: "bcf6e677-5fc9-447b-821c-f8ba0ecd65f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:34:26.676694 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.676691 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91602668-a9dd-48f5-87cb-7b7ce484db3b-home\" (UniqueName: \"kubernetes.io/empty-dir/91602668-a9dd-48f5-87cb-7b7ce484db3b-home\") pod \"91602668-a9dd-48f5-87cb-7b7ce484db3b\" (UID: \"91602668-a9dd-48f5-87cb-7b7ce484db3b\") " Nov 02 22:34:26.677032 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.676731 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4534e64e-1398-44a8-acb0-22db48ebea02-home\" (UniqueName: \"kubernetes.io/empty-dir/4534e64e-1398-44a8-acb0-22db48ebea02-home\") pod \"4534e64e-1398-44a8-acb0-22db48ebea02\" (UID: \"4534e64e-1398-44a8-acb0-22db48ebea02\") " Nov 02 22:34:26.677032 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.676778 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4534e64e-1398-44a8-acb0-22db48ebea02-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4534e64e-1398-44a8-acb0-22db48ebea02-dshm\") pod \"4534e64e-1398-44a8-acb0-22db48ebea02\" (UID: \"4534e64e-1398-44a8-acb0-22db48ebea02\") " Nov 02 22:34:26.677032 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.676790 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91602668-a9dd-48f5-87cb-7b7ce484db3b-model-cache" pod "91602668-a9dd-48f5-87cb-7b7ce484db3b" (UID: "91602668-a9dd-48f5-87cb-7b7ce484db3b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:34:26.677032 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.676811 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91602668-a9dd-48f5-87cb-7b7ce484db3b-tmp-dir" pod "91602668-a9dd-48f5-87cb-7b7ce484db3b" (UID: "91602668-a9dd-48f5-87cb-7b7ce484db3b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:34:26.677032 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.676816 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-model-cache\") pod \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\" (UID: \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\") " Nov 02 22:34:26.677032 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.676844 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-home" pod "bcf6e677-5fc9-447b-821c-f8ba0ecd65f8" (UID: "bcf6e677-5fc9-447b-821c-f8ba0ecd65f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:34:26.677032 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.676872 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4534e64e-1398-44a8-acb0-22db48ebea02-kube-api-access-ksmdg\" (UniqueName: \"kubernetes.io/projected/4534e64e-1398-44a8-acb0-22db48ebea02-kube-api-access-ksmdg\") pod \"4534e64e-1398-44a8-acb0-22db48ebea02\" (UID: \"4534e64e-1398-44a8-acb0-22db48ebea02\") " Nov 02 22:34:26.677032 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.676914 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91602668-a9dd-48f5-87cb-7b7ce484db3b-tls-certs\" (UniqueName: \"kubernetes.io/secret/91602668-a9dd-48f5-87cb-7b7ce484db3b-tls-certs\") pod \"91602668-a9dd-48f5-87cb-7b7ce484db3b\" (UID: \"91602668-a9dd-48f5-87cb-7b7ce484db3b\") " Nov 02 22:34:26.677032 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.676967 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4534e64e-1398-44a8-acb0-22db48ebea02-tls-certs\" (UniqueName: \"kubernetes.io/secret/4534e64e-1398-44a8-acb0-22db48ebea02-tls-certs\") pod \"4534e64e-1398-44a8-acb0-22db48ebea02\" (UID: \"4534e64e-1398-44a8-acb0-22db48ebea02\") " Nov 02 22:34:26.677032 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.677002 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4534e64e-1398-44a8-acb0-22db48ebea02-home" pod "4534e64e-1398-44a8-acb0-22db48ebea02" (UID: "4534e64e-1398-44a8-acb0-22db48ebea02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:34:26.677032 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.677006 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4534e64e-1398-44a8-acb0-22db48ebea02-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4534e64e-1398-44a8-acb0-22db48ebea02-model-cache\") pod \"4534e64e-1398-44a8-acb0-22db48ebea02\" (UID: \"4534e64e-1398-44a8-acb0-22db48ebea02\") " Nov 02 22:34:26.677720 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.677061 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-dshm\") pod \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\" (UID: \"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8\") " Nov 02 22:34:26.677720 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.677156 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-model-cache" pod "bcf6e677-5fc9-447b-821c-f8ba0ecd65f8" (UID: "bcf6e677-5fc9-447b-821c-f8ba0ecd65f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:34:26.677720 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.677174 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91602668-a9dd-48f5-87cb-7b7ce484db3b-kube-api-access-ds64t\" (UniqueName: \"kubernetes.io/projected/91602668-a9dd-48f5-87cb-7b7ce484db3b-kube-api-access-ds64t\") pod \"91602668-a9dd-48f5-87cb-7b7ce484db3b\" (UID: \"91602668-a9dd-48f5-87cb-7b7ce484db3b\") " Nov 02 22:34:26.677720 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.677217 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4534e64e-1398-44a8-acb0-22db48ebea02-model-cache" pod "4534e64e-1398-44a8-acb0-22db48ebea02" (UID: "4534e64e-1398-44a8-acb0-22db48ebea02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:34:26.677720 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.677351 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91602668-a9dd-48f5-87cb-7b7ce484db3b-home" pod "91602668-a9dd-48f5-87cb-7b7ce484db3b" (UID: "91602668-a9dd-48f5-87cb-7b7ce484db3b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:34:26.677720 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.677577 4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-tmp-dir\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:26.677720 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.677598 4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91602668-a9dd-48f5-87cb-7b7ce484db3b-model-cache\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:26.677720 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.677611 4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91602668-a9dd-48f5-87cb-7b7ce484db3b-tmp-dir\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:26.677720 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.677624 4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-home\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:26.677720 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.677639 4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91602668-a9dd-48f5-87cb-7b7ce484db3b-home\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:26.677720 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.677655 4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4534e64e-1398-44a8-acb0-22db48ebea02-home\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:26.677720 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.677673 4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-model-cache\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:26.677720 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.677686 4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4534e64e-1398-44a8-acb0-22db48ebea02-model-cache\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:26.677720 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.677573 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4534e64e-1398-44a8-acb0-22db48ebea02-tmp-dir" pod "4534e64e-1398-44a8-acb0-22db48ebea02" (UID: "4534e64e-1398-44a8-acb0-22db48ebea02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:34:26.679600 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.679572 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4534e64e-1398-44a8-acb0-22db48ebea02-tls-certs" pod "4534e64e-1398-44a8-acb0-22db48ebea02" (UID: "4534e64e-1398-44a8-acb0-22db48ebea02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:34:26.679713 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.679606 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-kube-api-access-2p7t8" pod "bcf6e677-5fc9-447b-821c-f8ba0ecd65f8" (UID: "bcf6e677-5fc9-447b-821c-f8ba0ecd65f8"). InnerVolumeSpecName "kube-api-access-2p7t8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:34:26.679830 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.679809 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4534e64e-1398-44a8-acb0-22db48ebea02-dshm" pod "4534e64e-1398-44a8-acb0-22db48ebea02" (UID: "4534e64e-1398-44a8-acb0-22db48ebea02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:34:26.679951 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.679928 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91602668-a9dd-48f5-87cb-7b7ce484db3b-kube-api-access-ds64t" pod "91602668-a9dd-48f5-87cb-7b7ce484db3b" (UID: "91602668-a9dd-48f5-87cb-7b7ce484db3b"). InnerVolumeSpecName "kube-api-access-ds64t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:34:26.680030 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.679998 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4534e64e-1398-44a8-acb0-22db48ebea02-kube-api-access-ksmdg" pod "4534e64e-1398-44a8-acb0-22db48ebea02" (UID: "4534e64e-1398-44a8-acb0-22db48ebea02"). InnerVolumeSpecName "kube-api-access-ksmdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:34:26.680271 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.680250 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-tls-certs" pod "bcf6e677-5fc9-447b-821c-f8ba0ecd65f8" (UID: "bcf6e677-5fc9-447b-821c-f8ba0ecd65f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:34:26.680601 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.680580 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91602668-a9dd-48f5-87cb-7b7ce484db3b-tls-certs" pod "91602668-a9dd-48f5-87cb-7b7ce484db3b" (UID: "91602668-a9dd-48f5-87cb-7b7ce484db3b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:34:26.680991 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.680973 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-dshm" pod "bcf6e677-5fc9-447b-821c-f8ba0ecd65f8" (UID: "bcf6e677-5fc9-447b-821c-f8ba0ecd65f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:34:26.681057 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.680983 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91602668-a9dd-48f5-87cb-7b7ce484db3b-dshm" pod "91602668-a9dd-48f5-87cb-7b7ce484db3b" (UID: "91602668-a9dd-48f5-87cb-7b7ce484db3b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:34:26.779018 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.778979 4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4534e64e-1398-44a8-acb0-22db48ebea02-dshm\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:26.779018 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.779019 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksmdg\" (UniqueName: \"kubernetes.io/projected/4534e64e-1398-44a8-acb0-22db48ebea02-kube-api-access-ksmdg\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:26.779299 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.779036 4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91602668-a9dd-48f5-87cb-7b7ce484db3b-tls-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:26.779299 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.779052 4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4534e64e-1398-44a8-acb0-22db48ebea02-tls-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:26.779299 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.779065 4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-dshm\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:26.779299 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.779077 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds64t\" (UniqueName: \"kubernetes.io/projected/91602668-a9dd-48f5-87cb-7b7ce484db3b-kube-api-access-ds64t\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:26.779299 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.779091 4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91602668-a9dd-48f5-87cb-7b7ce484db3b-dshm\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:26.779299 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.779106 4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4534e64e-1398-44a8-acb0-22db48ebea02-tmp-dir\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:26.779299 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.779121 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p7t8\" (UniqueName: \"kubernetes.io/projected/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-kube-api-access-2p7t8\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:26.779299 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:26.779151 4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8-tls-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:27.139789 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.139751 4935 generic.go:356] "Generic (PLEG): container finished" podID="4534e64e-1398-44a8-acb0-22db48ebea02" containerID="7cd7c96e3586a3fddbfeeee1c6cb10a14875ca6e47ad4625dd90e65329b64ed4" exitCode=0 Nov 02 22:34:27.139993 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.139811 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" Nov 02 22:34:27.139993 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.139838 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" event={"ID":"4534e64e-1398-44a8-acb0-22db48ebea02","Type":"ContainerDied","Data":"7cd7c96e3586a3fddbfeeee1c6cb10a14875ca6e47ad4625dd90e65329b64ed4"} Nov 02 22:34:27.139993 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.139880 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx" event={"ID":"4534e64e-1398-44a8-acb0-22db48ebea02","Type":"ContainerDied","Data":"5dd63669b94517ae7655ff4c8f45ff54a9f294fd6521c47b06fead03f769478d"} Nov 02 22:34:27.139993 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.139902 4935 scope.go:122] "RemoveContainer" containerID="7cd7c96e3586a3fddbfeeee1c6cb10a14875ca6e47ad4625dd90e65329b64ed4" Nov 02 22:34:27.141248 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.141224 4935 generic.go:356] "Generic (PLEG): container finished" podID="bcf6e677-5fc9-447b-821c-f8ba0ecd65f8" containerID="292db166abf313cb5281582a076f2403da95de31756a916f594c51b2cb90b16d" exitCode=0 Nov 02 22:34:27.141389 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.141367 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" Nov 02 22:34:27.141389 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.141351 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" event={"ID":"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8","Type":"ContainerDied","Data":"292db166abf313cb5281582a076f2403da95de31756a916f594c51b2cb90b16d"} Nov 02 22:34:27.141520 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.141411 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s" event={"ID":"bcf6e677-5fc9-447b-821c-f8ba0ecd65f8","Type":"ContainerDied","Data":"e77a2a43cf615dd52a37bf2b7347d39e5702b1d05a1fab4deae14ddb5633255f"} Nov 02 22:34:27.143030 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.143004 4935 generic.go:356] "Generic (PLEG): container finished" podID="91602668-a9dd-48f5-87cb-7b7ce484db3b" containerID="a488b0828abbb73269037cc2aec0ea7690e9339e6a1610f345e7371d15afcd6f" exitCode=0 Nov 02 22:34:27.143163 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.143118 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" event={"ID":"91602668-a9dd-48f5-87cb-7b7ce484db3b","Type":"ContainerDied","Data":"a488b0828abbb73269037cc2aec0ea7690e9339e6a1610f345e7371d15afcd6f"} Nov 02 22:34:27.143242 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.143179 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" event={"ID":"91602668-a9dd-48f5-87cb-7b7ce484db3b","Type":"ContainerDied","Data":"20e6807d3127049e976fd76f3f9516c43a3b22b3e5701e8819cda5f91e0e7aca"} Nov 02 22:34:27.143302 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.143236 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l" Nov 02 22:34:27.150659 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.150633 4935 scope.go:122] "RemoveContainer" containerID="7cd7c96e3586a3fddbfeeee1c6cb10a14875ca6e47ad4625dd90e65329b64ed4" Nov 02 22:34:27.151508 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:34:27.151483 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cd7c96e3586a3fddbfeeee1c6cb10a14875ca6e47ad4625dd90e65329b64ed4\": container with ID starting with 7cd7c96e3586a3fddbfeeee1c6cb10a14875ca6e47ad4625dd90e65329b64ed4 not found: ID does not exist" containerID="7cd7c96e3586a3fddbfeeee1c6cb10a14875ca6e47ad4625dd90e65329b64ed4" Nov 02 22:34:27.151667 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.151639 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cd7c96e3586a3fddbfeeee1c6cb10a14875ca6e47ad4625dd90e65329b64ed4"} err="failed to get container status \"7cd7c96e3586a3fddbfeeee1c6cb10a14875ca6e47ad4625dd90e65329b64ed4\": rpc error: code = NotFound desc = could not find container \"7cd7c96e3586a3fddbfeeee1c6cb10a14875ca6e47ad4625dd90e65329b64ed4\": container with ID starting with 7cd7c96e3586a3fddbfeeee1c6cb10a14875ca6e47ad4625dd90e65329b64ed4 not found: ID does not exist" Nov 02 22:34:27.151795 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.151669 4935 scope.go:122] "RemoveContainer" containerID="292db166abf313cb5281582a076f2403da95de31756a916f594c51b2cb90b16d" Nov 02 22:34:27.160697 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.160681 4935 scope.go:122] "RemoveContainer" containerID="292db166abf313cb5281582a076f2403da95de31756a916f594c51b2cb90b16d" Nov 02 22:34:27.160948 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:34:27.160926 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"292db166abf313cb5281582a076f2403da95de31756a916f594c51b2cb90b16d\": container with ID starting with 292db166abf313cb5281582a076f2403da95de31756a916f594c51b2cb90b16d not found: ID does not exist" containerID="292db166abf313cb5281582a076f2403da95de31756a916f594c51b2cb90b16d" Nov 02 22:34:27.161024 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.160957 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292db166abf313cb5281582a076f2403da95de31756a916f594c51b2cb90b16d"} err="failed to get container status \"292db166abf313cb5281582a076f2403da95de31756a916f594c51b2cb90b16d\": rpc error: code = NotFound desc = could not find container \"292db166abf313cb5281582a076f2403da95de31756a916f594c51b2cb90b16d\": container with ID starting with 292db166abf313cb5281582a076f2403da95de31756a916f594c51b2cb90b16d not found: ID does not exist" Nov 02 22:34:27.161024 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.160981 4935 scope.go:122] "RemoveContainer" containerID="a488b0828abbb73269037cc2aec0ea7690e9339e6a1610f345e7371d15afcd6f" Nov 02 22:34:27.169513 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.169496 4935 scope.go:122] "RemoveContainer" containerID="a488b0828abbb73269037cc2aec0ea7690e9339e6a1610f345e7371d15afcd6f" Nov 02 22:34:27.169789 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:34:27.169771 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a488b0828abbb73269037cc2aec0ea7690e9339e6a1610f345e7371d15afcd6f\": container with ID starting with a488b0828abbb73269037cc2aec0ea7690e9339e6a1610f345e7371d15afcd6f not found: ID does not exist" containerID="a488b0828abbb73269037cc2aec0ea7690e9339e6a1610f345e7371d15afcd6f" Nov 02 22:34:27.169862 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.169800 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a488b0828abbb73269037cc2aec0ea7690e9339e6a1610f345e7371d15afcd6f"} err="failed to get container status \"a488b0828abbb73269037cc2aec0ea7690e9339e6a1610f345e7371d15afcd6f\": rpc error: code = NotFound desc = could not find container \"a488b0828abbb73269037cc2aec0ea7690e9339e6a1610f345e7371d15afcd6f\": container with ID starting with a488b0828abbb73269037cc2aec0ea7690e9339e6a1610f345e7371d15afcd6f not found: ID does not exist" Nov 02 22:34:27.293919 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.293887 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx"] Nov 02 22:34:27.298506 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.298468 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f51151/distinct-2-w3-f51151-kserve-648bf5745b-xj7wx"] Nov 02 22:34:27.398770 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.398699 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l"] Nov 02 22:34:27.402467 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.402443 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f51151/distinct-w3-f51151-kserve-846c54f579-lfs2l"] Nov 02 22:34:27.503737 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.503705 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s"] Nov 02 22:34:27.505641 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.505613 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f51151/simulator-w3-f51151-kserve-869b5bf9bf-8jk2s"] Nov 02 22:34:27.776872 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.776842 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4534e64e-1398-44a8-acb0-22db48ebea02" path="/var/lib/kubelet/pods/4534e64e-1398-44a8-acb0-22db48ebea02/volumes" Nov 02 22:34:27.777454 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.777430 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91602668-a9dd-48f5-87cb-7b7ce484db3b" path="/var/lib/kubelet/pods/91602668-a9dd-48f5-87cb-7b7ce484db3b/volumes" Nov 02 22:34:27.777923 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:27.777907 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcf6e677-5fc9-447b-821c-f8ba0ecd65f8" path="/var/lib/kubelet/pods/bcf6e677-5fc9-447b-821c-f8ba0ecd65f8/volumes" Nov 02 22:34:28.468659 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:34:28.468622 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:34:33.967877 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:33.967840 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl"] Nov 02 22:34:33.968319 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:33.968082 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl" podUID="49454e6d-962a-4cc9-8976-4fdb2442bfdb" containerName="maas-api" containerID="cri-o://7bb067fe72bcc216d7deaceb79e29a24a695e26ae05c9610277a0073242895b2" gracePeriod=30 Nov 02 22:34:34.148193 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:34.147827 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:34.200761 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:34.200726 4935 generic.go:356] "Generic (PLEG): container finished" podID="49454e6d-962a-4cc9-8976-4fdb2442bfdb" containerID="7bb067fe72bcc216d7deaceb79e29a24a695e26ae05c9610277a0073242895b2" exitCode=0 Nov 02 22:34:34.200969 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:34.200845 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl" event={"ID":"49454e6d-962a-4cc9-8976-4fdb2442bfdb","Type":"ContainerDied","Data":"7bb067fe72bcc216d7deaceb79e29a24a695e26ae05c9610277a0073242895b2"} Nov 02 22:34:34.238681 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:34.238655 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl" Nov 02 22:34:34.359347 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:34.359271 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49454e6d-962a-4cc9-8976-4fdb2442bfdb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49454e6d-962a-4cc9-8976-4fdb2442bfdb-openshift-service-ca\") pod \"49454e6d-962a-4cc9-8976-4fdb2442bfdb\" (UID: \"49454e6d-962a-4cc9-8976-4fdb2442bfdb\") " Nov 02 22:34:34.359347 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:34.359331 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49454e6d-962a-4cc9-8976-4fdb2442bfdb-kube-api-access-fzsdk\" (UniqueName: \"kubernetes.io/projected/49454e6d-962a-4cc9-8976-4fdb2442bfdb-kube-api-access-fzsdk\") pod \"49454e6d-962a-4cc9-8976-4fdb2442bfdb\" (UID: \"49454e6d-962a-4cc9-8976-4fdb2442bfdb\") " Nov 02 22:34:34.359706 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:34.359400 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49454e6d-962a-4cc9-8976-4fdb2442bfdb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/49454e6d-962a-4cc9-8976-4fdb2442bfdb-metrics-certs\") pod \"49454e6d-962a-4cc9-8976-4fdb2442bfdb\" (UID: \"49454e6d-962a-4cc9-8976-4fdb2442bfdb\") " Nov 02 22:34:34.359706 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:34.359498 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49454e6d-962a-4cc9-8976-4fdb2442bfdb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49454e6d-962a-4cc9-8976-4fdb2442bfdb-maas-api-tls\") pod \"49454e6d-962a-4cc9-8976-4fdb2442bfdb\" (UID: \"49454e6d-962a-4cc9-8976-4fdb2442bfdb\") " Nov 02 22:34:34.362402 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:34.361983 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49454e6d-962a-4cc9-8976-4fdb2442bfdb-openshift-service-ca" pod "49454e6d-962a-4cc9-8976-4fdb2442bfdb" (UID: "49454e6d-962a-4cc9-8976-4fdb2442bfdb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:34:34.364337 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:34.364276 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49454e6d-962a-4cc9-8976-4fdb2442bfdb-maas-api-tls" pod "49454e6d-962a-4cc9-8976-4fdb2442bfdb" (UID: "49454e6d-962a-4cc9-8976-4fdb2442bfdb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:34:34.365901 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:34.365840 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49454e6d-962a-4cc9-8976-4fdb2442bfdb-kube-api-access-fzsdk" pod "49454e6d-962a-4cc9-8976-4fdb2442bfdb" (UID: "49454e6d-962a-4cc9-8976-4fdb2442bfdb"). InnerVolumeSpecName "kube-api-access-fzsdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:34:34.366369 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:34.366339 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49454e6d-962a-4cc9-8976-4fdb2442bfdb-metrics-certs" pod "49454e6d-962a-4cc9-8976-4fdb2442bfdb" (UID: "49454e6d-962a-4cc9-8976-4fdb2442bfdb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:34:34.460445 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:34.460414 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49454e6d-962a-4cc9-8976-4fdb2442bfdb-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:34.460445 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:34.460441 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49454e6d-962a-4cc9-8976-4fdb2442bfdb-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:34.460628 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:34.460450 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzsdk\" (UniqueName: \"kubernetes.io/projected/49454e6d-962a-4cc9-8976-4fdb2442bfdb-kube-api-access-fzsdk\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:34.460628 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:34.460462 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49454e6d-962a-4cc9-8976-4fdb2442bfdb-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:35.207367 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:35.207334 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl" Nov 02 22:34:35.207964 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:35.207363 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl" event={"ID":"49454e6d-962a-4cc9-8976-4fdb2442bfdb","Type":"ContainerDied","Data":"9c68629593899c834d0b667e1dd8574271619718388d6878a129b1fd3c91693c"} Nov 02 22:34:35.207964 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:35.207422 4935 scope.go:122] "RemoveContainer" containerID="7bb067fe72bcc216d7deaceb79e29a24a695e26ae05c9610277a0073242895b2" Nov 02 22:34:35.360097 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:35.360064 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl"] Nov 02 22:34:35.406061 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:35.406025 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f51151-59cd4686bf-4dwsl"] Nov 02 22:34:35.775472 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:35.775433 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49454e6d-962a-4cc9-8976-4fdb2442bfdb" path="/var/lib/kubelet/pods/49454e6d-962a-4cc9-8976-4fdb2442bfdb/volumes" Nov 02 22:34:38.773342 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:38.773251 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:38.977522 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:38.977490 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:46.322962 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:46.322916 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:46.824657 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:46.824622 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:48.233602 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:48.228071 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:48.437538 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:48.437498 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:48.829483 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:48.829419 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:51.240453 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:51.240395 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:51.717604 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:51.717572 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg"] Nov 02 22:34:51.717815 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:51.717792 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" podUID="9dae32e8-d70f-466c-8350-e1ae72e1372f" containerName="istio-proxy" containerID="cri-o://77ac75a5e3a1bc1f25b55ff07f9d183d89faf3ea7e471fc73312d2ad2c9f4728" gracePeriod=30 Nov 02 22:34:51.808918 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:51.808852 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:53.126634 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:53.126590 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-gh5gd_42f901f6-b299-4cc4-8871-2753cb497f94/manager/0.log" Nov 02 22:34:54.446831 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:54.446790 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:54.770099 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:54.770063 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:55.154262 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:55.154150 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-xbm86_88838f5d-567a-4e5e-9dc9-eff4470e53ff/manager/0.log" Nov 02 22:34:55.603276 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:55.603232 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-8ggc5_7797919e-d037-4b9c-9eb9-3d0d8af052fb/manager/0.log" Nov 02 22:34:56.984861 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:56.984801 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 22:34:57.066418 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.066386 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-istio-data\") pod \"9dae32e8-d70f-466c-8350-e1ae72e1372f\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " Nov 02 22:34:57.066604 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.066462 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dae32e8-d70f-466c-8350-e1ae72e1372f-istio-token\" (UniqueName: \"kubernetes.io/projected/9dae32e8-d70f-466c-8350-e1ae72e1372f-istio-token\") pod \"9dae32e8-d70f-466c-8350-e1ae72e1372f\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " Nov 02 22:34:57.066604 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.066527 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-workload-socket\") pod \"9dae32e8-d70f-466c-8350-e1ae72e1372f\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " Nov 02 22:34:57.066604 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.066557 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dae32e8-d70f-466c-8350-e1ae72e1372f-kube-api-access-bvnph\" (UniqueName: \"kubernetes.io/projected/9dae32e8-d70f-466c-8350-e1ae72e1372f-kube-api-access-bvnph\") pod \"9dae32e8-d70f-466c-8350-e1ae72e1372f\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " Nov 02 22:34:57.066823 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.066622 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-istio-envoy\") pod \"9dae32e8-d70f-466c-8350-e1ae72e1372f\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " Nov 02 22:34:57.066823 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.066649 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dae32e8-d70f-466c-8350-e1ae72e1372f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dae32e8-d70f-466c-8350-e1ae72e1372f-istiod-ca-cert\") pod \"9dae32e8-d70f-466c-8350-e1ae72e1372f\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " Nov 02 22:34:57.066823 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.066686 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9dae32e8-d70f-466c-8350-e1ae72e1372f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dae32e8-d70f-466c-8350-e1ae72e1372f-istio-podinfo\") pod \"9dae32e8-d70f-466c-8350-e1ae72e1372f\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " Nov 02 22:34:57.066823 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.066726 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-credential-socket\") pod \"9dae32e8-d70f-466c-8350-e1ae72e1372f\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " Nov 02 22:34:57.066823 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.066767 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-workload-certs\") pod \"9dae32e8-d70f-466c-8350-e1ae72e1372f\" (UID: \"9dae32e8-d70f-466c-8350-e1ae72e1372f\") " Nov 02 22:34:57.066823 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.066681 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-istio-data" pod "9dae32e8-d70f-466c-8350-e1ae72e1372f" (UID: "9dae32e8-d70f-466c-8350-e1ae72e1372f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:34:57.067126 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.066825 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-workload-socket" pod "9dae32e8-d70f-466c-8350-e1ae72e1372f" (UID: "9dae32e8-d70f-466c-8350-e1ae72e1372f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:34:57.067126 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.067096 4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-istio-data\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:57.067126 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.067115 4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-workload-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:57.067317 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.067111 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-credential-socket" pod "9dae32e8-d70f-466c-8350-e1ae72e1372f" (UID: "9dae32e8-d70f-466c-8350-e1ae72e1372f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:34:57.067317 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.067210 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-workload-certs" pod "9dae32e8-d70f-466c-8350-e1ae72e1372f" (UID: "9dae32e8-d70f-466c-8350-e1ae72e1372f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:34:57.067317 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.067305 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dae32e8-d70f-466c-8350-e1ae72e1372f-istiod-ca-cert" pod "9dae32e8-d70f-466c-8350-e1ae72e1372f" (UID: "9dae32e8-d70f-466c-8350-e1ae72e1372f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:34:57.069211 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.069179 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dae32e8-d70f-466c-8350-e1ae72e1372f-kube-api-access-bvnph" pod "9dae32e8-d70f-466c-8350-e1ae72e1372f" (UID: "9dae32e8-d70f-466c-8350-e1ae72e1372f"). InnerVolumeSpecName "kube-api-access-bvnph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:34:57.069374 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.069356 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dae32e8-d70f-466c-8350-e1ae72e1372f-istio-token" pod "9dae32e8-d70f-466c-8350-e1ae72e1372f" (UID: "9dae32e8-d70f-466c-8350-e1ae72e1372f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:34:57.069439 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.069403 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-istio-envoy" pod "9dae32e8-d70f-466c-8350-e1ae72e1372f" (UID: "9dae32e8-d70f-466c-8350-e1ae72e1372f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:34:57.069479 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.069453 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9dae32e8-d70f-466c-8350-e1ae72e1372f-istio-podinfo" pod "9dae32e8-d70f-466c-8350-e1ae72e1372f" (UID: "9dae32e8-d70f-466c-8350-e1ae72e1372f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:34:57.165484 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.165450 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-dhqhh_c6839ea0-0dae-40a5-a5c3-94220c3c70de/manager/0.log" Nov 02 22:34:57.168422 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.168398 4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dae32e8-d70f-466c-8350-e1ae72e1372f-istio-token\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:57.168544 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.168427 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvnph\" (UniqueName: \"kubernetes.io/projected/9dae32e8-d70f-466c-8350-e1ae72e1372f-kube-api-access-bvnph\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:57.168544 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.168443 4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-istio-envoy\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:57.168544 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.168455 4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dae32e8-d70f-466c-8350-e1ae72e1372f-istiod-ca-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:57.168544 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.168468 4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dae32e8-d70f-466c-8350-e1ae72e1372f-istio-podinfo\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:57.168544 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.168480 4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-credential-socket\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:57.168544 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.168493 4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dae32e8-d70f-466c-8350-e1ae72e1372f-workload-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:57.326084 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.326044 4935 generic.go:356] "Generic (PLEG): container finished" podID="9dae32e8-d70f-466c-8350-e1ae72e1372f" containerID="77ac75a5e3a1bc1f25b55ff07f9d183d89faf3ea7e471fc73312d2ad2c9f4728" exitCode=0 Nov 02 22:34:57.326280 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.326101 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" Nov 02 22:34:57.326280 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.326124 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" event={"ID":"9dae32e8-d70f-466c-8350-e1ae72e1372f","Type":"ContainerDied","Data":"77ac75a5e3a1bc1f25b55ff07f9d183d89faf3ea7e471fc73312d2ad2c9f4728"} Nov 02 22:34:57.326280 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.326207 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg" event={"ID":"9dae32e8-d70f-466c-8350-e1ae72e1372f","Type":"ContainerDied","Data":"52a7ed7fe0a13ea6edfe4213bde18d91387668340c41443f4a9c41c3eca5bc60"} Nov 02 22:34:57.326280 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.326238 4935 scope.go:122] "RemoveContainer" containerID="77ac75a5e3a1bc1f25b55ff07f9d183d89faf3ea7e471fc73312d2ad2c9f4728" Nov 02 22:34:57.335972 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.335953 4935 scope.go:122] "RemoveContainer" containerID="6ca7d14e9e01d80ac337890ce87c29db95be81f0082bd0914de1d08bb6d81fd4" Nov 02 22:34:57.344460 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.344436 4935 scope.go:122] "RemoveContainer" containerID="77ac75a5e3a1bc1f25b55ff07f9d183d89faf3ea7e471fc73312d2ad2c9f4728" Nov 02 22:34:57.344755 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:34:57.344733 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ac75a5e3a1bc1f25b55ff07f9d183d89faf3ea7e471fc73312d2ad2c9f4728\": container with ID starting with 77ac75a5e3a1bc1f25b55ff07f9d183d89faf3ea7e471fc73312d2ad2c9f4728 not found: ID does not exist" containerID="77ac75a5e3a1bc1f25b55ff07f9d183d89faf3ea7e471fc73312d2ad2c9f4728" Nov 02 22:34:57.344818 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.344763 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ac75a5e3a1bc1f25b55ff07f9d183d89faf3ea7e471fc73312d2ad2c9f4728"} err="failed to get container status \"77ac75a5e3a1bc1f25b55ff07f9d183d89faf3ea7e471fc73312d2ad2c9f4728\": rpc error: code = NotFound desc = could not find container \"77ac75a5e3a1bc1f25b55ff07f9d183d89faf3ea7e471fc73312d2ad2c9f4728\": container with ID starting with 77ac75a5e3a1bc1f25b55ff07f9d183d89faf3ea7e471fc73312d2ad2c9f4728 not found: ID does not exist" Nov 02 22:34:57.344818 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.344780 4935 scope.go:122] "RemoveContainer" containerID="6ca7d14e9e01d80ac337890ce87c29db95be81f0082bd0914de1d08bb6d81fd4" Nov 02 22:34:57.345038 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:34:57.345020 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ca7d14e9e01d80ac337890ce87c29db95be81f0082bd0914de1d08bb6d81fd4\": container with ID starting with 6ca7d14e9e01d80ac337890ce87c29db95be81f0082bd0914de1d08bb6d81fd4 not found: ID does not exist" containerID="6ca7d14e9e01d80ac337890ce87c29db95be81f0082bd0914de1d08bb6d81fd4" Nov 02 22:34:57.345086 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.345044 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ca7d14e9e01d80ac337890ce87c29db95be81f0082bd0914de1d08bb6d81fd4"} err="failed to get container status \"6ca7d14e9e01d80ac337890ce87c29db95be81f0082bd0914de1d08bb6d81fd4\": rpc error: code = NotFound desc = could not find container \"6ca7d14e9e01d80ac337890ce87c29db95be81f0082bd0914de1d08bb6d81fd4\": container with ID starting with 6ca7d14e9e01d80ac337890ce87c29db95be81f0082bd0914de1d08bb6d81fd4 not found: ID does not exist" Nov 02 22:34:57.394440 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.394418 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-28jss_cd2913d9-e59b-4b4a-b10a-a324e0c62f25/manager/0.log" Nov 02 22:34:57.478047 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.478013 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg"] Nov 02 22:34:57.481445 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.481416 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f51151-openshift-default-58588ffbdf-h99lg"] Nov 02 22:34:57.777549 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.777512 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dae32e8-d70f-466c-8350-e1ae72e1372f" path="/var/lib/kubelet/pods/9dae32e8-d70f-466c-8350-e1ae72e1372f/volumes" Nov 02 22:34:57.824221 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:57.824184 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d468bc7cc-hj64v_7e1298eb-5e6b-4e00-9269-d88a130d72c7/manager/1.log" Nov 02 22:34:58.468609 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:34:58.468572 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:34:58.520530 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:58.520491 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-nmwv9_0d2b5b50-c80c-4bac-8830-6c38da5fee20/proxy/0.log" Nov 02 22:34:59.179025 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:34:59.178992 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d468bc7cc-hj64v_7e1298eb-5e6b-4e00-9269-d88a130d72c7/manager/0.log" Nov 02 22:35:03.180984 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:35:03.180953 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:03.769749 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:35:03.769710 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:04.084644 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:35:04.084543 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:07.264945 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:35:07.264905 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:07.543021 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:35:07.542942 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:07.758312 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:35:07.758271 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:10.433079 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:35:10.433047 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:10.637277 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:35:10.637206 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:13.871141 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:35:13.871103 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:14.071921 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:35:14.071886 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:14.274312 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:35:14.274278 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:16.297502 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:35:16.297464 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:19.011544 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:35:19.011510 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:24.111341 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:35:24.111304 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:28.468243 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:35:28.468200 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:35:29.124434 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:35:29.124392 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:34.671663 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:35:34.671624 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:39.838120 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:35:39.838086 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:44.935848 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:35:44.935810 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:46.655058 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:35:46.655020 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:49.921156 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:35:49.921098 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:54.330053 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:35:54.330014 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:54.840384 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:35:54.840345 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:55.044702 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:35:55.044667 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:58.469555 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:35:58.469510 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:35:58.708097 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:35:58.708063 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:58.986882 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:35:58.986845 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:36:04.554645 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:04.554610 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:36:07.238158 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:07.238102 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:36:08.762222 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:08.762123 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d468bc7cc-hj64v"] Nov 02 22:36:08.762708 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:08.762393 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7d468bc7cc-hj64v" podUID="7e1298eb-5e6b-4e00-9269-d88a130d72c7" containerName="manager" containerID="cri-o://cb34b6a642ef2fb34cda353d348de41b942b42e57a3fe07f4f9fd9efe39ce643" gracePeriod=30 Nov 02 22:36:09.506602 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:09.506573 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d468bc7cc-hj64v" Nov 02 22:36:09.594923 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:09.594885 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e1298eb-5e6b-4e00-9269-d88a130d72c7-kube-api-access-6j7p7\" (UniqueName: \"kubernetes.io/projected/7e1298eb-5e6b-4e00-9269-d88a130d72c7-kube-api-access-6j7p7\") pod \"7e1298eb-5e6b-4e00-9269-d88a130d72c7\" (UID: \"7e1298eb-5e6b-4e00-9269-d88a130d72c7\") " Nov 02 22:36:09.595098 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:09.594970 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e1298eb-5e6b-4e00-9269-d88a130d72c7-webhook-cert\" (UniqueName: \"kubernetes.io/secret/7e1298eb-5e6b-4e00-9269-d88a130d72c7-webhook-cert\") pod \"7e1298eb-5e6b-4e00-9269-d88a130d72c7\" (UID: \"7e1298eb-5e6b-4e00-9269-d88a130d72c7\") " Nov 02 22:36:09.595098 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:09.595067 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e1298eb-5e6b-4e00-9269-d88a130d72c7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e1298eb-5e6b-4e00-9269-d88a130d72c7-metrics-certs\") pod \"7e1298eb-5e6b-4e00-9269-d88a130d72c7\" (UID: \"7e1298eb-5e6b-4e00-9269-d88a130d72c7\") " Nov 02 22:36:09.597122 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:09.597086 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e1298eb-5e6b-4e00-9269-d88a130d72c7-metrics-certs" pod "7e1298eb-5e6b-4e00-9269-d88a130d72c7" (UID: "7e1298eb-5e6b-4e00-9269-d88a130d72c7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:36:09.597249 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:09.597159 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1298eb-5e6b-4e00-9269-d88a130d72c7-kube-api-access-6j7p7" pod "7e1298eb-5e6b-4e00-9269-d88a130d72c7" (UID: "7e1298eb-5e6b-4e00-9269-d88a130d72c7"). InnerVolumeSpecName "kube-api-access-6j7p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:36:09.597249 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:09.597177 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e1298eb-5e6b-4e00-9269-d88a130d72c7-webhook-cert" pod "7e1298eb-5e6b-4e00-9269-d88a130d72c7" (UID: "7e1298eb-5e6b-4e00-9269-d88a130d72c7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:36:09.695864 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:09.695781 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e1298eb-5e6b-4e00-9269-d88a130d72c7-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:36:09.695864 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:09.695812 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j7p7\" (UniqueName: \"kubernetes.io/projected/7e1298eb-5e6b-4e00-9269-d88a130d72c7-kube-api-access-6j7p7\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:36:09.695864 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:09.695823 4935 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7e1298eb-5e6b-4e00-9269-d88a130d72c7-webhook-cert\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:36:09.704588 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:09.704555 4935 generic.go:356] "Generic (PLEG): container finished" podID="7e1298eb-5e6b-4e00-9269-d88a130d72c7" containerID="cb34b6a642ef2fb34cda353d348de41b942b42e57a3fe07f4f9fd9efe39ce643" exitCode=0 Nov 02 22:36:09.704736 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:09.704596 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d468bc7cc-hj64v" event={"ID":"7e1298eb-5e6b-4e00-9269-d88a130d72c7","Type":"ContainerDied","Data":"cb34b6a642ef2fb34cda353d348de41b942b42e57a3fe07f4f9fd9efe39ce643"} Nov 02 22:36:09.704736 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:09.704619 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d468bc7cc-hj64v" Nov 02 22:36:09.704736 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:09.704644 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d468bc7cc-hj64v" event={"ID":"7e1298eb-5e6b-4e00-9269-d88a130d72c7","Type":"ContainerDied","Data":"1981f9ba0661419c163b242827fcb7e0834eedee1e66c09c1e5438e813cf38ca"} Nov 02 22:36:09.704736 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:09.704666 4935 scope.go:122] "RemoveContainer" containerID="cb34b6a642ef2fb34cda353d348de41b942b42e57a3fe07f4f9fd9efe39ce643" Nov 02 22:36:09.714760 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:09.714738 4935 scope.go:122] "RemoveContainer" containerID="d04c62f90ac285517935481f026be68907815ec8e7c7f6c523bb29774be7ebc6" Nov 02 22:36:09.722403 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:09.722385 4935 scope.go:122] "RemoveContainer" containerID="cb34b6a642ef2fb34cda353d348de41b942b42e57a3fe07f4f9fd9efe39ce643" Nov 02 22:36:09.722675 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:36:09.722653 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb34b6a642ef2fb34cda353d348de41b942b42e57a3fe07f4f9fd9efe39ce643\": container with ID starting with cb34b6a642ef2fb34cda353d348de41b942b42e57a3fe07f4f9fd9efe39ce643 not found: ID does not exist" containerID="cb34b6a642ef2fb34cda353d348de41b942b42e57a3fe07f4f9fd9efe39ce643" Nov 02 22:36:09.722730 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:09.722683 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb34b6a642ef2fb34cda353d348de41b942b42e57a3fe07f4f9fd9efe39ce643"} err="failed to get container status \"cb34b6a642ef2fb34cda353d348de41b942b42e57a3fe07f4f9fd9efe39ce643\": rpc error: code = NotFound desc = could not find container \"cb34b6a642ef2fb34cda353d348de41b942b42e57a3fe07f4f9fd9efe39ce643\": container with ID starting with cb34b6a642ef2fb34cda353d348de41b942b42e57a3fe07f4f9fd9efe39ce643 not found: ID does not exist" Nov 02 22:36:09.722730 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:09.722701 4935 scope.go:122] "RemoveContainer" containerID="d04c62f90ac285517935481f026be68907815ec8e7c7f6c523bb29774be7ebc6" Nov 02 22:36:09.722930 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:36:09.722911 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d04c62f90ac285517935481f026be68907815ec8e7c7f6c523bb29774be7ebc6\": container with ID starting with d04c62f90ac285517935481f026be68907815ec8e7c7f6c523bb29774be7ebc6 not found: ID does not exist" containerID="d04c62f90ac285517935481f026be68907815ec8e7c7f6c523bb29774be7ebc6" Nov 02 22:36:09.722972 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:09.722936 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d04c62f90ac285517935481f026be68907815ec8e7c7f6c523bb29774be7ebc6"} err="failed to get container status \"d04c62f90ac285517935481f026be68907815ec8e7c7f6c523bb29774be7ebc6\": rpc error: code = NotFound desc = could not find container \"d04c62f90ac285517935481f026be68907815ec8e7c7f6c523bb29774be7ebc6\": container with ID starting with d04c62f90ac285517935481f026be68907815ec8e7c7f6c523bb29774be7ebc6 not found: ID does not exist" Nov 02 22:36:09.857323 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:09.857292 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d468bc7cc-hj64v"] Nov 02 22:36:09.861275 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:09.861249 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7d468bc7cc-hj64v"] Nov 02 22:36:11.776034 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:11.775988 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e1298eb-5e6b-4e00-9269-d88a130d72c7" path="/var/lib/kubelet/pods/7e1298eb-5e6b-4e00-9269-d88a130d72c7/volumes" Nov 02 22:36:28.468218 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:36:28.468178 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:36:46.995032 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:46.994991 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:36:49.517633 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:49.517597 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:36:52.096714 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:36:52.096674 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:36:58.467727 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:36:58.467686 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:37:28.467417 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:37:28.467378 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:37:43.106733 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.106646 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb"] Nov 02 22:37:43.107150 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107023 4935 cpu_manager.go:414] "containerMap: removing container" podUID="2d7d3443-7c43-4244-beda-b09241df09cd" containerName="istio-proxy" Nov 02 22:37:43.107150 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107034 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d7d3443-7c43-4244-beda-b09241df09cd" containerName="istio-proxy" Nov 02 22:37:43.107150 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107051 4935 cpu_manager.go:414] "containerMap: removing container" podUID="aab5db02-4890-45e1-bcf8-1c43c335a393" containerName="main" Nov 02 22:37:43.107150 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107059 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aab5db02-4890-45e1-bcf8-1c43c335a393" containerName="main" Nov 02 22:37:43.107150 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107069 4935 cpu_manager.go:414] "containerMap: removing container" podUID="cd425cbe-3140-4b2c-86c4-60f5261795f4" containerName="cleanup" Nov 02 22:37:43.107150 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107074 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd425cbe-3140-4b2c-86c4-60f5261795f4" containerName="cleanup" Nov 02 22:37:43.107150 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107081 4935 cpu_manager.go:414] "containerMap: removing container" podUID="58c2190d-dfa6-4bcd-8469-edd252023956" containerName="maas-api" Nov 02 22:37:43.107150 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107086 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58c2190d-dfa6-4bcd-8469-edd252023956" containerName="maas-api" Nov 02 22:37:43.107150 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107094 4935 cpu_manager.go:414] "containerMap: removing container" podUID="244817d6-8757-4aab-bfed-038ee1a69bb9" containerName="main" Nov 02 22:37:43.107150 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107099 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244817d6-8757-4aab-bfed-038ee1a69bb9" containerName="main" Nov 02 22:37:43.107150 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107107 4935 cpu_manager.go:414] "containerMap: removing container" podUID="cd425cbe-3140-4b2c-86c4-60f5261795f4" containerName="cleanup" Nov 02 22:37:43.107150 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107111 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd425cbe-3140-4b2c-86c4-60f5261795f4" containerName="cleanup" Nov 02 22:37:43.107150 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107121 4935 cpu_manager.go:414] "containerMap: removing container" podUID="49454e6d-962a-4cc9-8976-4fdb2442bfdb" containerName="maas-api" Nov 02 22:37:43.107150 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107127 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49454e6d-962a-4cc9-8976-4fdb2442bfdb" containerName="maas-api" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107165 4935 cpu_manager.go:414] "containerMap: removing container" podUID="91602668-a9dd-48f5-87cb-7b7ce484db3b" containerName="main" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107171 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91602668-a9dd-48f5-87cb-7b7ce484db3b" containerName="main" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107184 4935 cpu_manager.go:414] "containerMap: removing container" podUID="7cb3ac21-f737-4fc5-927e-4a8388582812" containerName="main" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107188 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb3ac21-f737-4fc5-927e-4a8388582812" containerName="main" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107198 4935 cpu_manager.go:414] "containerMap: removing container" podUID="7e1298eb-5e6b-4e00-9269-d88a130d72c7" containerName="manager" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107202 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e1298eb-5e6b-4e00-9269-d88a130d72c7" containerName="manager" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107209 4935 cpu_manager.go:414] "containerMap: removing container" podUID="bcf6e677-5fc9-447b-821c-f8ba0ecd65f8" containerName="main" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107215 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcf6e677-5fc9-447b-821c-f8ba0ecd65f8" containerName="main" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107227 4935 cpu_manager.go:414] "containerMap: removing container" podUID="7e1298eb-5e6b-4e00-9269-d88a130d72c7" containerName="manager" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107232 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e1298eb-5e6b-4e00-9269-d88a130d72c7" containerName="manager" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107238 4935 cpu_manager.go:414] "containerMap: removing container" podUID="9dae32e8-d70f-466c-8350-e1ae72e1372f" containerName="istio-proxy" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107243 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dae32e8-d70f-466c-8350-e1ae72e1372f" containerName="istio-proxy" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107249 4935 cpu_manager.go:414] "containerMap: removing container" podUID="095af893-e150-43b9-b479-263a270d05c4" containerName="main" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107254 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="095af893-e150-43b9-b479-263a270d05c4" containerName="main" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107264 4935 cpu_manager.go:414] "containerMap: removing container" podUID="3920de75-9c3e-4826-8de5-22fcc6d50b0e" containerName="main" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107268 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3920de75-9c3e-4826-8de5-22fcc6d50b0e" containerName="main" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107278 4935 cpu_manager.go:414] "containerMap: removing container" podUID="4534e64e-1398-44a8-acb0-22db48ebea02" containerName="main" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107283 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4534e64e-1398-44a8-acb0-22db48ebea02" containerName="main" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107290 4935 cpu_manager.go:414] "containerMap: removing container" podUID="ee1633c8-eaae-419f-8912-e506e14bd98d" containerName="maas-api" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107297 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee1633c8-eaae-419f-8912-e506e14bd98d" containerName="maas-api" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107368 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e1298eb-5e6b-4e00-9269-d88a130d72c7" containerName="manager" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107379 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcf6e677-5fc9-447b-821c-f8ba0ecd65f8" containerName="main" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107392 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e1298eb-5e6b-4e00-9269-d88a130d72c7" containerName="manager" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107400 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dae32e8-d70f-466c-8350-e1ae72e1372f" containerName="istio-proxy" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107410 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="095af893-e150-43b9-b479-263a270d05c4" containerName="main" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107418 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d7d3443-7c43-4244-beda-b09241df09cd" containerName="istio-proxy" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107428 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="3920de75-9c3e-4826-8de5-22fcc6d50b0e" containerName="main" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107437 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dae32e8-d70f-466c-8350-e1ae72e1372f" containerName="istio-proxy" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107444 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="4534e64e-1398-44a8-acb0-22db48ebea02" containerName="main" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107452 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee1633c8-eaae-419f-8912-e506e14bd98d" containerName="maas-api" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107459 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd425cbe-3140-4b2c-86c4-60f5261795f4" containerName="cleanup" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107468 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d7d3443-7c43-4244-beda-b09241df09cd" containerName="istio-proxy" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107475 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="aab5db02-4890-45e1-bcf8-1c43c335a393" containerName="main" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107483 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd425cbe-3140-4b2c-86c4-60f5261795f4" containerName="cleanup" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107492 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="58c2190d-dfa6-4bcd-8469-edd252023956" containerName="maas-api" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107500 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="244817d6-8757-4aab-bfed-038ee1a69bb9" containerName="main" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107507 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd425cbe-3140-4b2c-86c4-60f5261795f4" containerName="cleanup" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107516 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="49454e6d-962a-4cc9-8976-4fdb2442bfdb" containerName="maas-api" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107524 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="91602668-a9dd-48f5-87cb-7b7ce484db3b" containerName="main" Nov 02 22:37:43.107741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.107533 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb3ac21-f737-4fc5-927e-4a8388582812" containerName="main" Nov 02 22:37:43.110593 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.110570 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.223860 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.223824 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb"] Nov 02 22:37:43.230595 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.230566 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/371ba12b-5ead-4acc-aa5d-1de3d1e97b64-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hpgpb\" (UID: \"371ba12b-5ead-4acc-aa5d-1de3d1e97b64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.230757 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.230618 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371ba12b-5ead-4acc-aa5d-1de3d1e97b64-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hpgpb\" (UID: \"371ba12b-5ead-4acc-aa5d-1de3d1e97b64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.230757 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.230722 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371ba12b-5ead-4acc-aa5d-1de3d1e97b64-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hpgpb\" (UID: \"371ba12b-5ead-4acc-aa5d-1de3d1e97b64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.230887 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.230815 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371ba12b-5ead-4acc-aa5d-1de3d1e97b64-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hpgpb\" (UID: \"371ba12b-5ead-4acc-aa5d-1de3d1e97b64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.230887 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.230856 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm5bc\" (UniqueName: \"kubernetes.io/projected/371ba12b-5ead-4acc-aa5d-1de3d1e97b64-kube-api-access-tm5bc\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hpgpb\" (UID: \"371ba12b-5ead-4acc-aa5d-1de3d1e97b64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.230996 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.230912 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371ba12b-5ead-4acc-aa5d-1de3d1e97b64-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hpgpb\" (UID: \"371ba12b-5ead-4acc-aa5d-1de3d1e97b64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.231052 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.230996 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371ba12b-5ead-4acc-aa5d-1de3d1e97b64-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hpgpb\" (UID: \"371ba12b-5ead-4acc-aa5d-1de3d1e97b64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.231109 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.231049 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371ba12b-5ead-4acc-aa5d-1de3d1e97b64-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hpgpb\" (UID: \"371ba12b-5ead-4acc-aa5d-1de3d1e97b64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.231109 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.231089 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371ba12b-5ead-4acc-aa5d-1de3d1e97b64-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hpgpb\" (UID: \"371ba12b-5ead-4acc-aa5d-1de3d1e97b64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.332125 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.332089 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371ba12b-5ead-4acc-aa5d-1de3d1e97b64-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hpgpb\" (UID: \"371ba12b-5ead-4acc-aa5d-1de3d1e97b64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.332322 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.332146 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371ba12b-5ead-4acc-aa5d-1de3d1e97b64-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hpgpb\" (UID: \"371ba12b-5ead-4acc-aa5d-1de3d1e97b64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.332322 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.332181 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371ba12b-5ead-4acc-aa5d-1de3d1e97b64-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hpgpb\" (UID: \"371ba12b-5ead-4acc-aa5d-1de3d1e97b64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.332322 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.332209 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/371ba12b-5ead-4acc-aa5d-1de3d1e97b64-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hpgpb\" (UID: \"371ba12b-5ead-4acc-aa5d-1de3d1e97b64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.332322 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.332246 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371ba12b-5ead-4acc-aa5d-1de3d1e97b64-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hpgpb\" (UID: \"371ba12b-5ead-4acc-aa5d-1de3d1e97b64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.332545 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.332433 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371ba12b-5ead-4acc-aa5d-1de3d1e97b64-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hpgpb\" (UID: \"371ba12b-5ead-4acc-aa5d-1de3d1e97b64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.332545 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.332482 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371ba12b-5ead-4acc-aa5d-1de3d1e97b64-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hpgpb\" (UID: \"371ba12b-5ead-4acc-aa5d-1de3d1e97b64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.332545 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.332510 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm5bc\" (UniqueName: \"kubernetes.io/projected/371ba12b-5ead-4acc-aa5d-1de3d1e97b64-kube-api-access-tm5bc\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hpgpb\" (UID: \"371ba12b-5ead-4acc-aa5d-1de3d1e97b64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.332545 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.332544 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371ba12b-5ead-4acc-aa5d-1de3d1e97b64-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hpgpb\" (UID: \"371ba12b-5ead-4acc-aa5d-1de3d1e97b64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.332776 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.332615 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371ba12b-5ead-4acc-aa5d-1de3d1e97b64-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hpgpb\" (UID: \"371ba12b-5ead-4acc-aa5d-1de3d1e97b64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.332841 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.332768 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371ba12b-5ead-4acc-aa5d-1de3d1e97b64-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hpgpb\" (UID: \"371ba12b-5ead-4acc-aa5d-1de3d1e97b64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.332926 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.332894 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371ba12b-5ead-4acc-aa5d-1de3d1e97b64-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hpgpb\" (UID: \"371ba12b-5ead-4acc-aa5d-1de3d1e97b64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.332990 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.332966 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371ba12b-5ead-4acc-aa5d-1de3d1e97b64-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hpgpb\" (UID: \"371ba12b-5ead-4acc-aa5d-1de3d1e97b64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.333087 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.333064 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371ba12b-5ead-4acc-aa5d-1de3d1e97b64-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hpgpb\" (UID: \"371ba12b-5ead-4acc-aa5d-1de3d1e97b64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.334919 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.334887 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371ba12b-5ead-4acc-aa5d-1de3d1e97b64-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hpgpb\" (UID: \"371ba12b-5ead-4acc-aa5d-1de3d1e97b64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.335044 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.335025 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371ba12b-5ead-4acc-aa5d-1de3d1e97b64-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hpgpb\" (UID: \"371ba12b-5ead-4acc-aa5d-1de3d1e97b64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.385112 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.385031 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/371ba12b-5ead-4acc-aa5d-1de3d1e97b64-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hpgpb\" (UID: \"371ba12b-5ead-4acc-aa5d-1de3d1e97b64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.385344 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.385321 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm5bc\" (UniqueName: \"kubernetes.io/projected/371ba12b-5ead-4acc-aa5d-1de3d1e97b64-kube-api-access-tm5bc\") pod \"maas-default-gateway-openshift-default-fcb75c45f-hpgpb\" (UID: \"371ba12b-5ead-4acc-aa5d-1de3d1e97b64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.423614 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.423583 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:43.837776 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.837742 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb"] Nov 02 22:37:43.840739 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:37:43.840702 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371ba12b_5ead_4acc_aa5d_1de3d1e97b64.slice/crio-715c57e2d963430135fb98014b834923e431f2bf982a6c7484e2b4b6a4b018ed WatchSource:0}: Error finding container 715c57e2d963430135fb98014b834923e431f2bf982a6c7484e2b4b6a4b018ed: Status 404 returned error can't find the container with id 715c57e2d963430135fb98014b834923e431f2bf982a6c7484e2b4b6a4b018ed Nov 02 22:37:43.842714 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.842680 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:37:43.842832 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.842757 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:37:43.842832 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:43.842801 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:37:44.194424 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:44.194330 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" event={"ID":"371ba12b-5ead-4acc-aa5d-1de3d1e97b64","Type":"ContainerStarted","Data":"4741710a0cda5f5182b10fb602d18a7591003f550a6635fb32048cdb4f687969"} Nov 02 22:37:44.194424 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:44.194395 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" event={"ID":"371ba12b-5ead-4acc-aa5d-1de3d1e97b64","Type":"ContainerStarted","Data":"715c57e2d963430135fb98014b834923e431f2bf982a6c7484e2b4b6a4b018ed"} Nov 02 22:37:44.432156 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:44.424461 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:44.437160 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:44.433259 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-hpgpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 02 22:37:44.437160 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:44.433327 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" podUID="371ba12b-5ead-4acc-aa5d-1de3d1e97b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 02 22:37:45.436157 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:45.430531 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-hpgpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 02 22:37:45.436157 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:45.430596 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" podUID="371ba12b-5ead-4acc-aa5d-1de3d1e97b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 02 22:37:46.429168 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:46.428562 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-hpgpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 02 22:37:46.429168 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:46.428631 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" podUID="371ba12b-5ead-4acc-aa5d-1de3d1e97b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 02 22:37:47.350065 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:47.349980 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:37:47.430154 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:47.428549 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-hpgpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 02 22:37:47.430154 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:47.428629 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" podUID="371ba12b-5ead-4acc-aa5d-1de3d1e97b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 02 22:37:48.444156 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:48.435536 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-hpgpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 02 22:37:48.444156 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:48.435617 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" podUID="371ba12b-5ead-4acc-aa5d-1de3d1e97b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 02 22:37:49.427582 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:49.427539 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-hpgpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 02 22:37:49.427940 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:49.427897 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" podUID="371ba12b-5ead-4acc-aa5d-1de3d1e97b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 02 22:37:50.424146 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:50.424098 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-hpgpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 02 22:37:50.424646 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:50.424195 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" podUID="371ba12b-5ead-4acc-aa5d-1de3d1e97b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 02 22:37:52.425514 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:52.425471 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-hpgpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:37:52.426146 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:52.426088 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" podUID="371ba12b-5ead-4acc-aa5d-1de3d1e97b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:37:53.467302 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:53.467257 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-hpgpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:37:53.467760 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:53.467325 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" podUID="371ba12b-5ead-4acc-aa5d-1de3d1e97b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:37:54.348347 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:54.348316 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:54.348642 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:54.348603 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:54.349259 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:54.349238 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" Nov 02 22:37:54.504946 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:37:54.504837 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" podStartSLOduration=11.504824153 podStartE2EDuration="11.504824153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:37:44.31152645 +0000 UTC m=+5356.751101172" watchObservedRunningTime="2026-11-02 22:37:54.504824153 +0000 UTC m=+5366.944398875" Nov 02 22:37:58.468505 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:37:58.468462 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:38:06.709899 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:06.709865 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:38:13.287932 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.287896 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52"] Nov 02 22:38:13.288498 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.288477 4935 cpu_manager.go:414] "containerMap: removing container" podUID="9dae32e8-d70f-466c-8350-e1ae72e1372f" containerName="istio-proxy" Nov 02 22:38:13.288548 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.288502 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dae32e8-d70f-466c-8350-e1ae72e1372f" containerName="istio-proxy" Nov 02 22:38:13.288548 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.288541 4935 cpu_manager.go:414] "containerMap: removing container" podUID="2d7d3443-7c43-4244-beda-b09241df09cd" containerName="istio-proxy" Nov 02 22:38:13.288615 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.288550 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d7d3443-7c43-4244-beda-b09241df09cd" containerName="istio-proxy" Nov 02 22:38:13.288615 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.288568 4935 cpu_manager.go:414] "containerMap: removing container" podUID="cd425cbe-3140-4b2c-86c4-60f5261795f4" containerName="cleanup" Nov 02 22:38:13.288615 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.288576 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd425cbe-3140-4b2c-86c4-60f5261795f4" containerName="cleanup" Nov 02 22:38:13.292106 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.292077 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:13.381626 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.381579 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52"] Nov 02 22:38:13.407667 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.407630 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78539b8-fd5e-4fd6-a00d-95857df77139-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6fb52\" (UID: \"b78539b8-fd5e-4fd6-a00d-95857df77139\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:13.407864 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.407670 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78539b8-fd5e-4fd6-a00d-95857df77139-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6fb52\" (UID: \"b78539b8-fd5e-4fd6-a00d-95857df77139\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:13.407864 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.407760 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78539b8-fd5e-4fd6-a00d-95857df77139-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6fb52\" (UID: \"b78539b8-fd5e-4fd6-a00d-95857df77139\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:13.407864 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.407814 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78539b8-fd5e-4fd6-a00d-95857df77139-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6fb52\" (UID: \"b78539b8-fd5e-4fd6-a00d-95857df77139\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:13.407864 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.407841 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78539b8-fd5e-4fd6-a00d-95857df77139-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6fb52\" (UID: \"b78539b8-fd5e-4fd6-a00d-95857df77139\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:13.407864 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.407862 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78539b8-fd5e-4fd6-a00d-95857df77139-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6fb52\" (UID: \"b78539b8-fd5e-4fd6-a00d-95857df77139\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:13.408226 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.407880 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78539b8-fd5e-4fd6-a00d-95857df77139-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6fb52\" (UID: \"b78539b8-fd5e-4fd6-a00d-95857df77139\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:13.408226 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.408016 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78539b8-fd5e-4fd6-a00d-95857df77139-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6fb52\" (UID: \"b78539b8-fd5e-4fd6-a00d-95857df77139\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:13.408226 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.408069 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nrxz\" (UniqueName: \"kubernetes.io/projected/b78539b8-fd5e-4fd6-a00d-95857df77139-kube-api-access-8nrxz\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6fb52\" (UID: \"b78539b8-fd5e-4fd6-a00d-95857df77139\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:13.509362 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.509311 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78539b8-fd5e-4fd6-a00d-95857df77139-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6fb52\" (UID: \"b78539b8-fd5e-4fd6-a00d-95857df77139\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:13.509362 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.509368 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78539b8-fd5e-4fd6-a00d-95857df77139-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6fb52\" (UID: \"b78539b8-fd5e-4fd6-a00d-95857df77139\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:13.509641 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.509386 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78539b8-fd5e-4fd6-a00d-95857df77139-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6fb52\" (UID: \"b78539b8-fd5e-4fd6-a00d-95857df77139\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:13.509641 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.509405 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78539b8-fd5e-4fd6-a00d-95857df77139-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6fb52\" (UID: \"b78539b8-fd5e-4fd6-a00d-95857df77139\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:13.509641 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.509424 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78539b8-fd5e-4fd6-a00d-95857df77139-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6fb52\" (UID: \"b78539b8-fd5e-4fd6-a00d-95857df77139\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:13.509641 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.509441 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78539b8-fd5e-4fd6-a00d-95857df77139-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6fb52\" (UID: \"b78539b8-fd5e-4fd6-a00d-95857df77139\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:13.509641 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.509459 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78539b8-fd5e-4fd6-a00d-95857df77139-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6fb52\" (UID: \"b78539b8-fd5e-4fd6-a00d-95857df77139\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:13.509641 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.509488 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78539b8-fd5e-4fd6-a00d-95857df77139-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6fb52\" (UID: \"b78539b8-fd5e-4fd6-a00d-95857df77139\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:13.509641 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.509540 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nrxz\" (UniqueName: \"kubernetes.io/projected/b78539b8-fd5e-4fd6-a00d-95857df77139-kube-api-access-8nrxz\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6fb52\" (UID: \"b78539b8-fd5e-4fd6-a00d-95857df77139\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:13.509984 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.509794 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78539b8-fd5e-4fd6-a00d-95857df77139-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6fb52\" (UID: \"b78539b8-fd5e-4fd6-a00d-95857df77139\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:13.509984 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.509873 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78539b8-fd5e-4fd6-a00d-95857df77139-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6fb52\" (UID: \"b78539b8-fd5e-4fd6-a00d-95857df77139\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:13.510093 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.510069 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78539b8-fd5e-4fd6-a00d-95857df77139-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6fb52\" (UID: \"b78539b8-fd5e-4fd6-a00d-95857df77139\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:13.510184 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.510166 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78539b8-fd5e-4fd6-a00d-95857df77139-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6fb52\" (UID: \"b78539b8-fd5e-4fd6-a00d-95857df77139\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:13.510264 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.510243 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78539b8-fd5e-4fd6-a00d-95857df77139-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6fb52\" (UID: \"b78539b8-fd5e-4fd6-a00d-95857df77139\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:13.512419 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.512388 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78539b8-fd5e-4fd6-a00d-95857df77139-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6fb52\" (UID: \"b78539b8-fd5e-4fd6-a00d-95857df77139\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:13.512598 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.512582 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78539b8-fd5e-4fd6-a00d-95857df77139-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6fb52\" (UID: \"b78539b8-fd5e-4fd6-a00d-95857df77139\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:13.562779 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.562701 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78539b8-fd5e-4fd6-a00d-95857df77139-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6fb52\" (UID: \"b78539b8-fd5e-4fd6-a00d-95857df77139\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:13.562912 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.562825 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nrxz\" (UniqueName: \"kubernetes.io/projected/b78539b8-fd5e-4fd6-a00d-95857df77139-kube-api-access-8nrxz\") pod \"maas-default-gateway-openshift-default-fcb75c45f-6fb52\" (UID: \"b78539b8-fd5e-4fd6-a00d-95857df77139\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:13.605579 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:13.605536 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:14.023004 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:14.022968 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52"] Nov 02 22:38:14.026321 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:38:14.026287 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78539b8_fd5e_4fd6_a00d_95857df77139.slice/crio-52e4702cea45116da6e8b3913504fe363c130b46d2c3df79b27b901d0f2fe79a WatchSource:0}: Error finding container 52e4702cea45116da6e8b3913504fe363c130b46d2c3df79b27b901d0f2fe79a: Status 404 returned error can't find the container with id 52e4702cea45116da6e8b3913504fe363c130b46d2c3df79b27b901d0f2fe79a Nov 02 22:38:14.029187 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:14.029155 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:38:14.029274 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:14.029229 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:38:14.029274 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:14.029259 4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:38:14.357754 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:14.357649 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" event={"ID":"b78539b8-fd5e-4fd6-a00d-95857df77139","Type":"ContainerStarted","Data":"516e18d0fe359f3d7c667ad2a1f2cbc5888c08b91b537ede699bd65d2f658d0b"} Nov 02 22:38:14.357754 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:14.357721 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" event={"ID":"b78539b8-fd5e-4fd6-a00d-95857df77139","Type":"ContainerStarted","Data":"52e4702cea45116da6e8b3913504fe363c130b46d2c3df79b27b901d0f2fe79a"} Nov 02 22:38:14.606246 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:14.606211 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:15.607730 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:15.607690 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6fb52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 22:38:15.608164 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:15.607755 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" podUID="b78539b8-fd5e-4fd6-a00d-95857df77139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 22:38:16.606794 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:16.606750 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6fb52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 22:38:16.607013 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:16.606820 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" podUID="b78539b8-fd5e-4fd6-a00d-95857df77139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 22:38:17.606617 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:17.606579 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6fb52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 22:38:17.607024 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:17.606648 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" podUID="b78539b8-fd5e-4fd6-a00d-95857df77139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 22:38:18.606515 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:18.606470 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6fb52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 22:38:18.606712 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:18.606536 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" podUID="b78539b8-fd5e-4fd6-a00d-95857df77139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 22:38:19.606684 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:19.606646 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6fb52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 22:38:19.606868 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:19.606701 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" podUID="b78539b8-fd5e-4fd6-a00d-95857df77139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 22:38:20.606983 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:20.606942 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6fb52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 22:38:20.607451 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:20.607002 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" podUID="b78539b8-fd5e-4fd6-a00d-95857df77139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 22:38:21.606205 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:21.606173 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-6fb52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 22:38:21.606470 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:21.606229 4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" podUID="b78539b8-fd5e-4fd6-a00d-95857df77139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 22:38:22.610840 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:22.610807 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:22.611299 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:22.611087 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:22.611812 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:22.611796 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" Nov 02 22:38:22.813415 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:22.813362 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-6fb52" podStartSLOduration=9.813347605 podStartE2EDuration="9.813347605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:38:14.469826115 +0000 UTC m=+5386.909400842" watchObservedRunningTime="2026-11-02 22:38:22.813347605 +0000 UTC m=+5395.252922326" Nov 02 22:38:28.470992 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:38:28.470947 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:38:28.717889 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:28.717861 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 22:38:28.726804 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:28.726748 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 22:38:28.758295 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:28.758272 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 22:38:28.767033 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:28.767012 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 22:38:43.722016 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:43.721971 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54"] Nov 02 22:38:43.734234 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:43.733340 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" Nov 02 22:38:43.808029 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:43.807991 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54"] Nov 02 22:38:43.821982 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:43.821945 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 22:38:43.821982 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:43.821949 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4dfx7\"" Nov 02 22:38:43.822236 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:43.822081 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 22:38:43.835413 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:43.835373 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-58e3ef14-kserve-self-signed-certs\"" Nov 02 22:38:43.836234 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:43.836192 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7bxt\" (UniqueName: \"kubernetes.io/projected/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-kube-api-access-n7bxt\") pod \"e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54\" (UID: \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\") " pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" Nov 02 22:38:43.836554 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:43.836531 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-home\") pod \"e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54\" (UID: \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\") " pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" Nov 02 22:38:43.836790 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:43.836769 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-tls-certs\") pod \"e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54\" (UID: \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\") " pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" Nov 02 22:38:43.836998 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:43.836979 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-tmp-dir\") pod \"e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54\" (UID: \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\") " pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" Nov 02 22:38:43.837243 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:43.837224 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-model-cache\") pod \"e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54\" (UID: \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\") " pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" Nov 02 22:38:43.837441 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:43.837422 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-dshm\") pod \"e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54\" (UID: \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\") " pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" Nov 02 22:38:43.913820 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:43.913786 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:38:43.948171 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:43.946648 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-home\") pod \"e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54\" (UID: \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\") " pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" Nov 02 22:38:43.948171 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:43.946727 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-tls-certs\") pod \"e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54\" (UID: \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\") " pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" Nov 02 22:38:43.948171 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:43.946765 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-tmp-dir\") pod \"e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54\" (UID: \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\") " pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" Nov 02 22:38:43.948171 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:43.946824 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-model-cache\") pod \"e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54\" (UID: \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\") " pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" Nov 02 22:38:43.948171 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:43.946848 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-dshm\") pod \"e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54\" (UID: \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\") " pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" Nov 02 22:38:43.948171 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:43.946911 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7bxt\" (UniqueName: \"kubernetes.io/projected/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-kube-api-access-n7bxt\") pod \"e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54\" (UID: \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\") " pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" Nov 02 22:38:43.948171 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:43.947737 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-home\") pod \"e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54\" (UID: \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\") " pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" Nov 02 22:38:43.990053 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:43.949469 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-model-cache\") pod \"e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54\" (UID: \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\") " pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" Nov 02 22:38:43.990053 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:43.949737 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-tmp-dir\") pod \"e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54\" (UID: \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\") " pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" Nov 02 22:38:43.990053 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:43.957750 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-tls-certs\") pod \"e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54\" (UID: \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\") " pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" Nov 02 22:38:43.990053 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:43.979444 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-dshm\") pod \"e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54\" (UID: \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\") " pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" Nov 02 22:38:44.007202 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:44.007162 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7bxt\" (UniqueName: \"kubernetes.io/projected/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-kube-api-access-n7bxt\") pod \"e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54\" (UID: \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\") " pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" Nov 02 22:38:44.092790 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:44.092346 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" Nov 02 22:38:44.562164 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:44.561810 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54"] Nov 02 22:38:45.559980 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:45.559944 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" event={"ID":"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f","Type":"ContainerStarted","Data":"12d65a43405c80abd487e905aa3e8f5221bd4e6be89dcb5ad82523f131c808b9"} Nov 02 22:38:45.560581 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:45.560009 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" event={"ID":"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f","Type":"ContainerStarted","Data":"f9cec2c7bd6bc9456bde1bf6807d248ec351785b6239f6244803d95e81fcebcc"} Nov 02 22:38:45.629451 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:45.629403 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:38:45.668954 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:45.668896 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" podStartSLOduration=2.668881585 podStartE2EDuration="2.668881585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:38:45.667050315 +0000 UTC m=+5418.106625037" watchObservedRunningTime="2026-11-02 22:38:45.668881585 +0000 UTC m=+5418.108456307" Nov 02 22:38:46.093292 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:46.093256 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" Nov 02 22:38:47.671340 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:47.671297 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:38:53.539637 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:53.539597 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:38:54.093030 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:54.092990 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" Nov 02 22:38:54.105811 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:54.105781 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" Nov 02 22:38:54.642028 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:38:54.641998 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" Nov 02 22:38:58.468661 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:38:58.468621 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:39:28.468712 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:39:28.468673 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:39:48.003516 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:39:48.003481 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:51.715855 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:39:51.715814 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54"] Nov 02 22:39:51.799653 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:39:51.799593 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:52.011715 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:39:52.011622 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:58.519302 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:39:58.519257 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:40:06.741834 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:06.741768 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" podUID="0a2b4f6f-03e9-4cea-a0b8-11347f79b78f" containerName="main" containerID="cri-o://12d65a43405c80abd487e905aa3e8f5221bd4e6be89dcb5ad82523f131c808b9" gracePeriod=45 Nov 02 22:40:06.994991 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:06.994921 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" Nov 02 22:40:07.051741 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:07.051706 4935 generic.go:356] "Generic (PLEG): container finished" podID="0a2b4f6f-03e9-4cea-a0b8-11347f79b78f" containerID="12d65a43405c80abd487e905aa3e8f5221bd4e6be89dcb5ad82523f131c808b9" exitCode=0 Nov 02 22:40:07.051890 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:07.051747 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" Nov 02 22:40:07.051890 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:07.051784 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" event={"ID":"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f","Type":"ContainerDied","Data":"12d65a43405c80abd487e905aa3e8f5221bd4e6be89dcb5ad82523f131c808b9"} Nov 02 22:40:07.051890 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:07.051828 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54" event={"ID":"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f","Type":"ContainerDied","Data":"f9cec2c7bd6bc9456bde1bf6807d248ec351785b6239f6244803d95e81fcebcc"} Nov 02 22:40:07.051890 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:07.051845 4935 scope.go:122] "RemoveContainer" containerID="12d65a43405c80abd487e905aa3e8f5221bd4e6be89dcb5ad82523f131c808b9" Nov 02 22:40:07.061002 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:07.060983 4935 scope.go:122] "RemoveContainer" containerID="12d65a43405c80abd487e905aa3e8f5221bd4e6be89dcb5ad82523f131c808b9" Nov 02 22:40:07.061318 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:40:07.061299 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12d65a43405c80abd487e905aa3e8f5221bd4e6be89dcb5ad82523f131c808b9\": container with ID starting with 12d65a43405c80abd487e905aa3e8f5221bd4e6be89dcb5ad82523f131c808b9 not found: ID does not exist" containerID="12d65a43405c80abd487e905aa3e8f5221bd4e6be89dcb5ad82523f131c808b9" Nov 02 22:40:07.061389 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:07.061327 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12d65a43405c80abd487e905aa3e8f5221bd4e6be89dcb5ad82523f131c808b9"} err="failed to get container status \"12d65a43405c80abd487e905aa3e8f5221bd4e6be89dcb5ad82523f131c808b9\": rpc error: code = NotFound desc = could not find container \"12d65a43405c80abd487e905aa3e8f5221bd4e6be89dcb5ad82523f131c808b9\": container with ID starting with 12d65a43405c80abd487e905aa3e8f5221bd4e6be89dcb5ad82523f131c808b9 not found: ID does not exist" Nov 02 22:40:07.072760 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:07.072738 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-tls-certs\") pod \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\" (UID: \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\") " Nov 02 22:40:07.072854 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:07.072781 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-dshm\") pod \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\" (UID: \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\") " Nov 02 22:40:07.072854 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:07.072826 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-tmp-dir\") pod \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\" (UID: \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\") " Nov 02 22:40:07.072940 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:07.072871 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-kube-api-access-n7bxt\" (UniqueName: \"kubernetes.io/projected/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-kube-api-access-n7bxt\") pod \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\" (UID: \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\") " Nov 02 22:40:07.072940 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:07.072893 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-model-cache\") pod \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\" (UID: \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\") " Nov 02 22:40:07.073049 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:07.072954 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-home\" (UniqueName: \"kubernetes.io/empty-dir/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-home\") pod \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\" (UID: \"0a2b4f6f-03e9-4cea-a0b8-11347f79b78f\") " Nov 02 22:40:07.073175 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:07.073148 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-tmp-dir" pod "0a2b4f6f-03e9-4cea-a0b8-11347f79b78f" (UID: "0a2b4f6f-03e9-4cea-a0b8-11347f79b78f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:40:07.073336 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:07.073264 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-model-cache" pod "0a2b4f6f-03e9-4cea-a0b8-11347f79b78f" (UID: "0a2b4f6f-03e9-4cea-a0b8-11347f79b78f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:40:07.073336 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:07.073323 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-home" pod "0a2b4f6f-03e9-4cea-a0b8-11347f79b78f" (UID: "0a2b4f6f-03e9-4cea-a0b8-11347f79b78f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:40:07.073515 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:07.073413 4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-tmp-dir\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:40:07.073515 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:07.073432 4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-model-cache\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:40:07.073515 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:07.073445 4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-home\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:40:07.075124 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:07.075094 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-dshm" pod "0a2b4f6f-03e9-4cea-a0b8-11347f79b78f" (UID: "0a2b4f6f-03e9-4cea-a0b8-11347f79b78f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:40:07.075124 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:07.075112 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-kube-api-access-n7bxt" pod "0a2b4f6f-03e9-4cea-a0b8-11347f79b78f" (UID: "0a2b4f6f-03e9-4cea-a0b8-11347f79b78f"). InnerVolumeSpecName "kube-api-access-n7bxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:40:07.075307 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:07.075217 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-tls-certs" pod "0a2b4f6f-03e9-4cea-a0b8-11347f79b78f" (UID: "0a2b4f6f-03e9-4cea-a0b8-11347f79b78f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:40:07.174801 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:07.174767 4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-tls-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:40:07.174801 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:07.174796 4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-dshm\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:40:07.174801 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:07.174807 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7bxt\" (UniqueName: \"kubernetes.io/projected/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f-kube-api-access-n7bxt\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:40:07.464539 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:07.464504 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54"] Nov 02 22:40:07.466468 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:07.466442 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-58e3ef14-kserve-c9bf8456-krg54"] Nov 02 22:40:07.775459 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:07.775424 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a2b4f6f-03e9-4cea-a0b8-11347f79b78f" path="/var/lib/kubelet/pods/0a2b4f6f-03e9-4cea-a0b8-11347f79b78f/volumes" Nov 02 22:40:09.512542 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:09.512501 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:10.087165 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:10.087103 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:10.388182 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:10.388080 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:14.291722 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:14.291689 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:14.565825 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:14.565738 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:17.427887 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:17.427852 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:17.638684 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:17.638642 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:20.361682 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:20.361645 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq"] Nov 02 22:40:20.362102 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:20.362073 4935 cpu_manager.go:414] "containerMap: removing container" podUID="0a2b4f6f-03e9-4cea-a0b8-11347f79b78f" containerName="main" Nov 02 22:40:20.362102 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:20.362095 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a2b4f6f-03e9-4cea-a0b8-11347f79b78f" containerName="main" Nov 02 22:40:20.362243 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:20.362215 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a2b4f6f-03e9-4cea-a0b8-11347f79b78f" containerName="main" Nov 02 22:40:20.365373 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:20.365350 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq" Nov 02 22:40:20.396626 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:20.396424 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5de6998f-1144-4c32-994e-b88f5f378dd4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq\" (UID: \"5de6998f-1144-4c32-994e-b88f5f378dd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq" Nov 02 22:40:20.397521 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:20.397470 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5de6998f-1144-4c32-994e-b88f5f378dd4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq\" (UID: \"5de6998f-1144-4c32-994e-b88f5f378dd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq" Nov 02 22:40:20.397832 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:20.397805 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhhfg\" (UniqueName: \"kubernetes.io/projected/5de6998f-1144-4c32-994e-b88f5f378dd4-kube-api-access-vhhfg\") pod \"maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq\" (UID: \"5de6998f-1144-4c32-994e-b88f5f378dd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq" Nov 02 22:40:20.397981 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:20.397963 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5de6998f-1144-4c32-994e-b88f5f378dd4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq\" (UID: \"5de6998f-1144-4c32-994e-b88f5f378dd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq" Nov 02 22:40:20.461947 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:20.461910 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-6faf18-serving-cert\"" Nov 02 22:40:20.464267 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:20.464240 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq"] Nov 02 22:40:20.498791 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:20.498759 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhhfg\" (UniqueName: \"kubernetes.io/projected/5de6998f-1144-4c32-994e-b88f5f378dd4-kube-api-access-vhhfg\") pod \"maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq\" (UID: \"5de6998f-1144-4c32-994e-b88f5f378dd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq" Nov 02 22:40:20.498960 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:20.498890 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5de6998f-1144-4c32-994e-b88f5f378dd4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq\" (UID: \"5de6998f-1144-4c32-994e-b88f5f378dd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq" Nov 02 22:40:20.498960 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:20.498932 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5de6998f-1144-4c32-994e-b88f5f378dd4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq\" (UID: \"5de6998f-1144-4c32-994e-b88f5f378dd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq" Nov 02 22:40:20.499082 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:20.498975 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5de6998f-1144-4c32-994e-b88f5f378dd4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq\" (UID: \"5de6998f-1144-4c32-994e-b88f5f378dd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq" Nov 02 22:40:20.499769 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:20.499741 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5de6998f-1144-4c32-994e-b88f5f378dd4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq\" (UID: \"5de6998f-1144-4c32-994e-b88f5f378dd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq" Nov 02 22:40:20.500286 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:40:20.500082 4935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-serving-cert: secret "maas-api-e2e-worker-single-main-6faf18-serving-cert" not found Nov 02 22:40:20.500403 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:40:20.500343 4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5de6998f-1144-4c32-994e-b88f5f378dd4-maas-api-tls podName:5de6998f-1144-4c32-994e-b88f5f378dd4 nodeName:}" failed. No retries permitted until 2026-11-02 22:40:21.00032194 +0000 UTC m=+5513.439896654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5de6998f-1144-4c32-994e-b88f5f378dd4-maas-api-tls") pod "maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq" (UID: "5de6998f-1144-4c32-994e-b88f5f378dd4") : secret "maas-api-e2e-worker-single-main-6faf18-serving-cert" not found Nov 02 22:40:20.502599 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:20.502567 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5de6998f-1144-4c32-994e-b88f5f378dd4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq\" (UID: \"5de6998f-1144-4c32-994e-b88f5f378dd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq" Nov 02 22:40:20.553727 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:20.553691 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhhfg\" (UniqueName: \"kubernetes.io/projected/5de6998f-1144-4c32-994e-b88f5f378dd4-kube-api-access-vhhfg\") pod \"maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq\" (UID: \"5de6998f-1144-4c32-994e-b88f5f378dd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq" Nov 02 22:40:20.643932 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:20.643840 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:20.863040 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:20.863003 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:21.003332 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:21.003249 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5de6998f-1144-4c32-994e-b88f5f378dd4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq\" (UID: \"5de6998f-1144-4c32-994e-b88f5f378dd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq" Nov 02 22:40:21.005673 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:21.005639 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5de6998f-1144-4c32-994e-b88f5f378dd4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq\" (UID: \"5de6998f-1144-4c32-994e-b88f5f378dd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq" Nov 02 22:40:21.278058 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:21.278021 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq" Nov 02 22:40:21.646500 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:21.646465 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq"] Nov 02 22:40:21.648707 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:40:21.648678 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de6998f_1144_4c32_994e_b88f5f378dd4.slice/crio-3719ad023c710c646f9427c1ddb9d06d6ceee041916b6f7eb723d1774242c088 WatchSource:0}: Error finding container 3719ad023c710c646f9427c1ddb9d06d6ceee041916b6f7eb723d1774242c088: Status 404 returned error can't find the container with id 3719ad023c710c646f9427c1ddb9d06d6ceee041916b6f7eb723d1774242c088 Nov 02 22:40:21.650100 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:21.650078 4935 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:40:22.152106 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:22.152070 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq" event={"ID":"5de6998f-1144-4c32-994e-b88f5f378dd4","Type":"ContainerStarted","Data":"3719ad023c710c646f9427c1ddb9d06d6ceee041916b6f7eb723d1774242c088"} Nov 02 22:40:23.158933 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:23.158894 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq" event={"ID":"5de6998f-1144-4c32-994e-b88f5f378dd4","Type":"ContainerStarted","Data":"44d3bd7b5665073c530a406096f19b5f3f9de7c50d7602666e9f822cdfc3aa38"} Nov 02 22:40:23.159384 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:23.158993 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq" Nov 02 22:40:23.267313 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:23.267253 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq" podStartSLOduration=2.311484531 podStartE2EDuration="3.267237417s" totalImagesPullingTime="955.752886ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:40:23.265866024 +0000 UTC m=+5515.705440746" watchObservedRunningTime="2026-11-02 22:40:23.267237417 +0000 UTC m=+5515.706812196" Nov 02 22:40:25.698722 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:25.698685 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:28.468307 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:40:28.468262 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:40:29.171916 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:29.171883 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq" Nov 02 22:40:30.610645 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:30.610612 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:35.500526 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:35.500492 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:39.027863 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:39.027764 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:39.583718 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:39.583670 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:39.811544 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:39.811497 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:40.803164 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:40.803112 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:42.428666 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:42.428628 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:45.706558 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:45.706518 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:48.333899 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:48.333849 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:50.604333 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:50.604289 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:55.487359 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:55.487325 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:58.467829 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:40:58.467791 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:40:58.956884 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:40:58.956843 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:00.391971 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:00.391931 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:05.221679 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:05.221643 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:10.098860 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:10.098823 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:15.007214 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:15.007176 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:19.895512 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:19.895404 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:24.704115 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:24.704082 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:28.468919 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:41:28.468875 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:41:29.620181 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:29.620120 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:34.506746 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:34.506714 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:38.296400 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:38.296359 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:38.729043 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:38.728933 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:39.396733 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:39.396695 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:43.330051 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:43.330014 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:44.315038 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:44.315003 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:44.553264 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:44.553225 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:45.676750 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:45.676712 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:45.917013 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:45.916977 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:48.660382 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:48.660341 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:48.814414 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:48.814378 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq"] Nov 02 22:41:48.814625 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:48.814606 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq" podUID="5de6998f-1144-4c32-994e-b88f5f378dd4" containerName="maas-api" containerID="cri-o://44d3bd7b5665073c530a406096f19b5f3f9de7c50d7602666e9f822cdfc3aa38" gracePeriod=30 Nov 02 22:41:49.007158 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:49.007030 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:49.165598 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:49.165559 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq" podUID="5de6998f-1144-4c32-994e-b88f5f378dd4" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.155:8443/health\": dial tcp 10.132.0.155:8443: connect: connection refused" Nov 02 22:41:49.938464 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:49.938416 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:50.070436 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:50.070410 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq" Nov 02 22:41:50.111448 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:50.111419 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5de6998f-1144-4c32-994e-b88f5f378dd4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5de6998f-1144-4c32-994e-b88f5f378dd4-maas-api-tls\") pod \"5de6998f-1144-4c32-994e-b88f5f378dd4\" (UID: \"5de6998f-1144-4c32-994e-b88f5f378dd4\") " Nov 02 22:41:50.111586 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:50.111457 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5de6998f-1144-4c32-994e-b88f5f378dd4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5de6998f-1144-4c32-994e-b88f5f378dd4-metrics-certs\") pod \"5de6998f-1144-4c32-994e-b88f5f378dd4\" (UID: \"5de6998f-1144-4c32-994e-b88f5f378dd4\") " Nov 02 22:41:50.111586 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:50.111506 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5de6998f-1144-4c32-994e-b88f5f378dd4-kube-api-access-vhhfg\" (UniqueName: \"kubernetes.io/projected/5de6998f-1144-4c32-994e-b88f5f378dd4-kube-api-access-vhhfg\") pod \"5de6998f-1144-4c32-994e-b88f5f378dd4\" (UID: \"5de6998f-1144-4c32-994e-b88f5f378dd4\") " Nov 02 22:41:50.111586 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:50.111539 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5de6998f-1144-4c32-994e-b88f5f378dd4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5de6998f-1144-4c32-994e-b88f5f378dd4-openshift-service-ca\") pod \"5de6998f-1144-4c32-994e-b88f5f378dd4\" (UID: \"5de6998f-1144-4c32-994e-b88f5f378dd4\") " Nov 02 22:41:50.111959 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:50.111937 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5de6998f-1144-4c32-994e-b88f5f378dd4-openshift-service-ca" pod "5de6998f-1144-4c32-994e-b88f5f378dd4" (UID: "5de6998f-1144-4c32-994e-b88f5f378dd4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:41:50.113500 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:50.113476 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5de6998f-1144-4c32-994e-b88f5f378dd4-maas-api-tls" pod "5de6998f-1144-4c32-994e-b88f5f378dd4" (UID: "5de6998f-1144-4c32-994e-b88f5f378dd4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:41:50.113709 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:50.113688 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de6998f-1144-4c32-994e-b88f5f378dd4-kube-api-access-vhhfg" pod "5de6998f-1144-4c32-994e-b88f5f378dd4" (UID: "5de6998f-1144-4c32-994e-b88f5f378dd4"). InnerVolumeSpecName "kube-api-access-vhhfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:41:50.113709 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:50.113696 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5de6998f-1144-4c32-994e-b88f5f378dd4-metrics-certs" pod "5de6998f-1144-4c32-994e-b88f5f378dd4" (UID: "5de6998f-1144-4c32-994e-b88f5f378dd4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:41:50.212907 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:50.212822 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhhfg\" (UniqueName: \"kubernetes.io/projected/5de6998f-1144-4c32-994e-b88f5f378dd4-kube-api-access-vhhfg\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:41:50.212907 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:50.212859 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5de6998f-1144-4c32-994e-b88f5f378dd4-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:41:50.212907 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:50.212874 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5de6998f-1144-4c32-994e-b88f5f378dd4-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:41:50.212907 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:50.212883 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5de6998f-1144-4c32-994e-b88f5f378dd4-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:41:50.613893 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:50.613855 4935 generic.go:356] "Generic (PLEG): container finished" podID="5de6998f-1144-4c32-994e-b88f5f378dd4" containerID="44d3bd7b5665073c530a406096f19b5f3f9de7c50d7602666e9f822cdfc3aa38" exitCode=0 Nov 02 22:41:50.614113 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:50.613905 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq" Nov 02 22:41:50.614113 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:50.613945 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq" event={"ID":"5de6998f-1144-4c32-994e-b88f5f378dd4","Type":"ContainerDied","Data":"44d3bd7b5665073c530a406096f19b5f3f9de7c50d7602666e9f822cdfc3aa38"} Nov 02 22:41:50.614113 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:50.614000 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq" event={"ID":"5de6998f-1144-4c32-994e-b88f5f378dd4","Type":"ContainerDied","Data":"3719ad023c710c646f9427c1ddb9d06d6ceee041916b6f7eb723d1774242c088"} Nov 02 22:41:50.614113 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:50.614018 4935 scope.go:122] "RemoveContainer" containerID="44d3bd7b5665073c530a406096f19b5f3f9de7c50d7602666e9f822cdfc3aa38" Nov 02 22:41:50.623950 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:50.623927 4935 scope.go:122] "RemoveContainer" containerID="44d3bd7b5665073c530a406096f19b5f3f9de7c50d7602666e9f822cdfc3aa38" Nov 02 22:41:50.624219 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:41:50.624198 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44d3bd7b5665073c530a406096f19b5f3f9de7c50d7602666e9f822cdfc3aa38\": container with ID starting with 44d3bd7b5665073c530a406096f19b5f3f9de7c50d7602666e9f822cdfc3aa38 not found: ID does not exist" containerID="44d3bd7b5665073c530a406096f19b5f3f9de7c50d7602666e9f822cdfc3aa38" Nov 02 22:41:50.624303 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:50.624240 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44d3bd7b5665073c530a406096f19b5f3f9de7c50d7602666e9f822cdfc3aa38"} err="failed to get container status \"44d3bd7b5665073c530a406096f19b5f3f9de7c50d7602666e9f822cdfc3aa38\": rpc error: code = NotFound desc = could not find container \"44d3bd7b5665073c530a406096f19b5f3f9de7c50d7602666e9f822cdfc3aa38\": container with ID starting with 44d3bd7b5665073c530a406096f19b5f3f9de7c50d7602666e9f822cdfc3aa38 not found: ID does not exist" Nov 02 22:41:50.774618 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:50.774579 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq"] Nov 02 22:41:50.820826 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:50.820793 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6faf18-8566874cf8-btwfq"] Nov 02 22:41:51.775960 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:51.775926 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5de6998f-1144-4c32-994e-b88f5f378dd4" path="/var/lib/kubelet/pods/5de6998f-1144-4c32-994e-b88f5f378dd4/volumes" Nov 02 22:41:55.348369 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:55.348335 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:55.564384 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:41:55.564349 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:58.468290 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:41:58.468253 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:42:19.361491 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:19.361382 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:42:20.238304 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:20.238267 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:42:20.718216 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:20.718163 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:42:20.954801 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:20.954765 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:42:22.447975 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:22.447942 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:42:22.878879 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:22.878836 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:42:23.468771 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:23.468718 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:42:25.893277 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:25.893243 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:42:26.110569 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:26.110526 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:42:28.469212 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:42:28.469173 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:42:39.068276 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.068235 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs"] Nov 02 22:42:39.068866 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.068843 4935 cpu_manager.go:414] "containerMap: removing container" podUID="5de6998f-1144-4c32-994e-b88f5f378dd4" containerName="maas-api" Nov 02 22:42:39.068954 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.068868 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5de6998f-1144-4c32-994e-b88f5f378dd4" containerName="maas-api" Nov 02 22:42:39.069048 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.069035 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="5de6998f-1144-4c32-994e-b88f5f378dd4" containerName="maas-api" Nov 02 22:42:39.073394 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.073371 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" Nov 02 22:42:39.122746 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.122693 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 22:42:39.168823 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.168794 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-51a7601a-kserve-self-signed-certs\"" Nov 02 22:42:39.169014 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.168908 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4dfx7\"" Nov 02 22:42:39.169014 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.168912 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 22:42:39.171074 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.171050 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs"] Nov 02 22:42:39.181987 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.181962 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc1708b-8a37-4594-a73a-33edbe600fa2-dshm\") pod \"e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs\" (UID: \"dbc1708b-8a37-4594-a73a-33edbe600fa2\") " pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" Nov 02 22:42:39.182102 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.182015 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc1708b-8a37-4594-a73a-33edbe600fa2-tls-certs\") pod \"e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs\" (UID: \"dbc1708b-8a37-4594-a73a-33edbe600fa2\") " pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" Nov 02 22:42:39.182165 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.182142 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc1708b-8a37-4594-a73a-33edbe600fa2-model-cache\") pod \"e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs\" (UID: \"dbc1708b-8a37-4594-a73a-33edbe600fa2\") " pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" Nov 02 22:42:39.182221 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.182175 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc1708b-8a37-4594-a73a-33edbe600fa2-tmp-dir\") pod \"e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs\" (UID: \"dbc1708b-8a37-4594-a73a-33edbe600fa2\") " pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" Nov 02 22:42:39.182221 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.182215 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwxj4\" (UniqueName: \"kubernetes.io/projected/dbc1708b-8a37-4594-a73a-33edbe600fa2-kube-api-access-pwxj4\") pod \"e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs\" (UID: \"dbc1708b-8a37-4594-a73a-33edbe600fa2\") " pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" Nov 02 22:42:39.182315 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.182298 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbc1708b-8a37-4594-a73a-33edbe600fa2-home\") pod \"e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs\" (UID: \"dbc1708b-8a37-4594-a73a-33edbe600fa2\") " pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" Nov 02 22:42:39.272461 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.272424 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:42:39.283071 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.283040 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc1708b-8a37-4594-a73a-33edbe600fa2-model-cache\") pod \"e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs\" (UID: \"dbc1708b-8a37-4594-a73a-33edbe600fa2\") " pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" Nov 02 22:42:39.283254 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.283073 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc1708b-8a37-4594-a73a-33edbe600fa2-tmp-dir\") pod \"e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs\" (UID: \"dbc1708b-8a37-4594-a73a-33edbe600fa2\") " pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" Nov 02 22:42:39.283254 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.283112 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwxj4\" (UniqueName: \"kubernetes.io/projected/dbc1708b-8a37-4594-a73a-33edbe600fa2-kube-api-access-pwxj4\") pod \"e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs\" (UID: \"dbc1708b-8a37-4594-a73a-33edbe600fa2\") " pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" Nov 02 22:42:39.283254 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.283180 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbc1708b-8a37-4594-a73a-33edbe600fa2-home\") pod \"e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs\" (UID: \"dbc1708b-8a37-4594-a73a-33edbe600fa2\") " pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" Nov 02 22:42:39.283254 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.283228 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc1708b-8a37-4594-a73a-33edbe600fa2-dshm\") pod \"e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs\" (UID: \"dbc1708b-8a37-4594-a73a-33edbe600fa2\") " pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" Nov 02 22:42:39.283475 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.283257 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc1708b-8a37-4594-a73a-33edbe600fa2-tls-certs\") pod \"e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs\" (UID: \"dbc1708b-8a37-4594-a73a-33edbe600fa2\") " pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" Nov 02 22:42:39.283543 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.283513 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc1708b-8a37-4594-a73a-33edbe600fa2-model-cache\") pod \"e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs\" (UID: \"dbc1708b-8a37-4594-a73a-33edbe600fa2\") " pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" Nov 02 22:42:39.283602 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.283567 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc1708b-8a37-4594-a73a-33edbe600fa2-tmp-dir\") pod \"e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs\" (UID: \"dbc1708b-8a37-4594-a73a-33edbe600fa2\") " pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" Nov 02 22:42:39.283684 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.283655 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbc1708b-8a37-4594-a73a-33edbe600fa2-home\") pod \"e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs\" (UID: \"dbc1708b-8a37-4594-a73a-33edbe600fa2\") " pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" Nov 02 22:42:39.285597 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.285563 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc1708b-8a37-4594-a73a-33edbe600fa2-dshm\") pod \"e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs\" (UID: \"dbc1708b-8a37-4594-a73a-33edbe600fa2\") " pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" Nov 02 22:42:39.285759 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.285739 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc1708b-8a37-4594-a73a-33edbe600fa2-tls-certs\") pod \"e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs\" (UID: \"dbc1708b-8a37-4594-a73a-33edbe600fa2\") " pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" Nov 02 22:42:39.337510 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.337438 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwxj4\" (UniqueName: \"kubernetes.io/projected/dbc1708b-8a37-4594-a73a-33edbe600fa2-kube-api-access-pwxj4\") pod \"e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs\" (UID: \"dbc1708b-8a37-4594-a73a-33edbe600fa2\") " pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" Nov 02 22:42:39.396765 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.396730 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" Nov 02 22:42:39.777204 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.777013 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs"] Nov 02 22:42:39.779863 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:42:39.779833 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbc1708b_8a37_4594_a73a_33edbe600fa2.slice/crio-58ea281a5b2ea2774d5a540daaa11daa4d0198e6abdf75863e747646bf4e986f WatchSource:0}: Error finding container 58ea281a5b2ea2774d5a540daaa11daa4d0198e6abdf75863e747646bf4e986f: Status 404 returned error can't find the container with id 58ea281a5b2ea2774d5a540daaa11daa4d0198e6abdf75863e747646bf4e986f Nov 02 22:42:39.876923 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.876891 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" event={"ID":"dbc1708b-8a37-4594-a73a-33edbe600fa2","Type":"ContainerStarted","Data":"1ba7d826900764bc758fd4bd63c0c51615280189d0b36cc6ec8d8dd54f90737b"} Nov 02 22:42:39.877071 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.876936 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" event={"ID":"dbc1708b-8a37-4594-a73a-33edbe600fa2","Type":"ContainerStarted","Data":"58ea281a5b2ea2774d5a540daaa11daa4d0198e6abdf75863e747646bf4e986f"} Nov 02 22:42:39.983797 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:39.983734 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" podStartSLOduration=0.983716662 podStartE2EDuration="983.716662ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:42:39.982867298 +0000 UTC m=+5652.422442022" watchObservedRunningTime="2026-11-02 22:42:39.983716662 +0000 UTC m=+5652.423291384" Nov 02 22:42:40.397360 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:40.397328 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" Nov 02 22:42:48.976683 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:48.976646 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:42:49.397284 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:49.397239 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" Nov 02 22:42:49.416604 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:49.416571 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" Nov 02 22:42:49.940736 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:42:49.940704 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" Nov 02 22:42:58.467674 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:42:58.467632 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:43:02.382708 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:02.382662 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:28.471388 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:43:28.471344 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:43:28.763353 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:28.763318 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 22:43:28.775241 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:28.775203 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 22:43:28.804611 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:28.804584 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 22:43:28.817894 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:28.817866 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 22:43:31.787181 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:31.787142 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:32.001597 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:32.001554 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:38.620603 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:38.620520 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs"] Nov 02 22:43:38.745328 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:38.745260 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:49.325403 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:49.325322 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:53.641756 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:53.641688 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" podUID="dbc1708b-8a37-4594-a73a-33edbe600fa2" containerName="main" containerID="cri-o://1ba7d826900764bc758fd4bd63c0c51615280189d0b36cc6ec8d8dd54f90737b" gracePeriod=45 Nov 02 22:43:53.910587 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:53.910562 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" Nov 02 22:43:54.064352 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:54.064321 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbc1708b-8a37-4594-a73a-33edbe600fa2-home\" (UniqueName: \"kubernetes.io/empty-dir/dbc1708b-8a37-4594-a73a-33edbe600fa2-home\") pod \"dbc1708b-8a37-4594-a73a-33edbe600fa2\" (UID: \"dbc1708b-8a37-4594-a73a-33edbe600fa2\") " Nov 02 22:43:54.064352 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:54.064357 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbc1708b-8a37-4594-a73a-33edbe600fa2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc1708b-8a37-4594-a73a-33edbe600fa2-dshm\") pod \"dbc1708b-8a37-4594-a73a-33edbe600fa2\" (UID: \"dbc1708b-8a37-4594-a73a-33edbe600fa2\") " Nov 02 22:43:54.064586 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:54.064411 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbc1708b-8a37-4594-a73a-33edbe600fa2-tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc1708b-8a37-4594-a73a-33edbe600fa2-tls-certs\") pod \"dbc1708b-8a37-4594-a73a-33edbe600fa2\" (UID: \"dbc1708b-8a37-4594-a73a-33edbe600fa2\") " Nov 02 22:43:54.064586 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:54.064435 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbc1708b-8a37-4594-a73a-33edbe600fa2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc1708b-8a37-4594-a73a-33edbe600fa2-tmp-dir\") pod \"dbc1708b-8a37-4594-a73a-33edbe600fa2\" (UID: \"dbc1708b-8a37-4594-a73a-33edbe600fa2\") " Nov 02 22:43:54.064586 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:54.064506 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbc1708b-8a37-4594-a73a-33edbe600fa2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc1708b-8a37-4594-a73a-33edbe600fa2-model-cache\") pod \"dbc1708b-8a37-4594-a73a-33edbe600fa2\" (UID: \"dbc1708b-8a37-4594-a73a-33edbe600fa2\") " Nov 02 22:43:54.064586 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:54.064580 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbc1708b-8a37-4594-a73a-33edbe600fa2-kube-api-access-pwxj4\" (UniqueName: \"kubernetes.io/projected/dbc1708b-8a37-4594-a73a-33edbe600fa2-kube-api-access-pwxj4\") pod \"dbc1708b-8a37-4594-a73a-33edbe600fa2\" (UID: \"dbc1708b-8a37-4594-a73a-33edbe600fa2\") " Nov 02 22:43:54.064819 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:54.064638 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc1708b-8a37-4594-a73a-33edbe600fa2-home" pod "dbc1708b-8a37-4594-a73a-33edbe600fa2" (UID: "dbc1708b-8a37-4594-a73a-33edbe600fa2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:43:54.064819 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:54.064799 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc1708b-8a37-4594-a73a-33edbe600fa2-tmp-dir" pod "dbc1708b-8a37-4594-a73a-33edbe600fa2" (UID: "dbc1708b-8a37-4594-a73a-33edbe600fa2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:43:54.064920 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:54.064825 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc1708b-8a37-4594-a73a-33edbe600fa2-model-cache" pod "dbc1708b-8a37-4594-a73a-33edbe600fa2" (UID: "dbc1708b-8a37-4594-a73a-33edbe600fa2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:43:54.064963 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:54.064928 4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbc1708b-8a37-4594-a73a-33edbe600fa2-home\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:43:54.065032 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:54.064971 4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc1708b-8a37-4594-a73a-33edbe600fa2-tmp-dir\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:43:54.065032 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:54.064983 4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc1708b-8a37-4594-a73a-33edbe600fa2-model-cache\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:43:54.066685 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:54.066662 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc1708b-8a37-4594-a73a-33edbe600fa2-dshm" pod "dbc1708b-8a37-4594-a73a-33edbe600fa2" (UID: "dbc1708b-8a37-4594-a73a-33edbe600fa2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:43:54.066785 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:54.066698 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbc1708b-8a37-4594-a73a-33edbe600fa2-kube-api-access-pwxj4" pod "dbc1708b-8a37-4594-a73a-33edbe600fa2" (UID: "dbc1708b-8a37-4594-a73a-33edbe600fa2"). InnerVolumeSpecName "kube-api-access-pwxj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:43:54.066785 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:54.066747 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbc1708b-8a37-4594-a73a-33edbe600fa2-tls-certs" pod "dbc1708b-8a37-4594-a73a-33edbe600fa2" (UID: "dbc1708b-8a37-4594-a73a-33edbe600fa2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:43:54.166318 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:54.166234 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwxj4\" (UniqueName: \"kubernetes.io/projected/dbc1708b-8a37-4594-a73a-33edbe600fa2-kube-api-access-pwxj4\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:43:54.166318 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:54.166271 4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc1708b-8a37-4594-a73a-33edbe600fa2-dshm\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:43:54.166318 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:54.166280 4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc1708b-8a37-4594-a73a-33edbe600fa2-tls-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:43:54.346772 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:54.346738 4935 generic.go:356] "Generic (PLEG): container finished" podID="dbc1708b-8a37-4594-a73a-33edbe600fa2" containerID="1ba7d826900764bc758fd4bd63c0c51615280189d0b36cc6ec8d8dd54f90737b" exitCode=0 Nov 02 22:43:54.346988 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:54.346796 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" Nov 02 22:43:54.346988 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:54.346828 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" event={"ID":"dbc1708b-8a37-4594-a73a-33edbe600fa2","Type":"ContainerDied","Data":"1ba7d826900764bc758fd4bd63c0c51615280189d0b36cc6ec8d8dd54f90737b"} Nov 02 22:43:54.346988 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:54.346878 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs" event={"ID":"dbc1708b-8a37-4594-a73a-33edbe600fa2","Type":"ContainerDied","Data":"58ea281a5b2ea2774d5a540daaa11daa4d0198e6abdf75863e747646bf4e986f"} Nov 02 22:43:54.346988 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:54.346901 4935 scope.go:122] "RemoveContainer" containerID="1ba7d826900764bc758fd4bd63c0c51615280189d0b36cc6ec8d8dd54f90737b" Nov 02 22:43:54.357012 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:54.356994 4935 scope.go:122] "RemoveContainer" containerID="1ba7d826900764bc758fd4bd63c0c51615280189d0b36cc6ec8d8dd54f90737b" Nov 02 22:43:54.357283 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:43:54.357257 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba7d826900764bc758fd4bd63c0c51615280189d0b36cc6ec8d8dd54f90737b\": container with ID starting with 1ba7d826900764bc758fd4bd63c0c51615280189d0b36cc6ec8d8dd54f90737b not found: ID does not exist" containerID="1ba7d826900764bc758fd4bd63c0c51615280189d0b36cc6ec8d8dd54f90737b" Nov 02 22:43:54.357403 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:54.357291 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba7d826900764bc758fd4bd63c0c51615280189d0b36cc6ec8d8dd54f90737b"} err="failed to get container status \"1ba7d826900764bc758fd4bd63c0c51615280189d0b36cc6ec8d8dd54f90737b\": rpc error: code = NotFound desc = could not find container \"1ba7d826900764bc758fd4bd63c0c51615280189d0b36cc6ec8d8dd54f90737b\": container with ID starting with 1ba7d826900764bc758fd4bd63c0c51615280189d0b36cc6ec8d8dd54f90737b not found: ID does not exist" Nov 02 22:43:54.498121 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:54.498050 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs"] Nov 02 22:43:54.544215 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:54.544183 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-51a7601a-kserve-58b8f7c785-86rbs"] Nov 02 22:43:55.775249 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:55.775215 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbc1708b-8a37-4594-a73a-33edbe600fa2" path="/var/lib/kubelet/pods/dbc1708b-8a37-4594-a73a-33edbe600fa2/volumes" Nov 02 22:43:55.967298 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:43:55.967263 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:58.469033 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:43:58.468915 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:44:28.468061 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:44:28.468025 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:44:35.231355 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:44:35.231316 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:44:49.639844 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:44:49.639805 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:44:58.468556 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:44:58.468517 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:45:13.695850 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:45:13.695767 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:45:26.447633 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:45:26.447601 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:45:26.885838 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:45:26.885798 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:45:28.468115 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:45:28.468079 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:45:29.402559 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:45:29.402520 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:45:46.310720 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:45:46.310682 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:45:49.997356 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:45:49.997323 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:45:58.468729 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:45:58.468687 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:46:28.468161 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:46:28.468099 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:46:50.297185 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:46:50.297076 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:46:50.524485 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:46:50.524455 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:46:58.467876 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:46:58.467829 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:47:17.121157 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:47:17.121095 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:47:17.360115 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:47:17.360074 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:47:17.585724 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:47:17.585690 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:47:25.389213 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:47:25.389121 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-hpgpb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:47:25.389903 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:47:25.389265 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" podUID="371ba12b-5ead-4acc-aa5d-1de3d1e97b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:47:28.469399 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:47:28.469357 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:47:50.641357 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:47:50.641321 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:47:58.512039 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:47:58.511999 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:48:28.468029 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:48:28.467988 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:48:28.804482 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:48:28.804454 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 22:48:28.820480 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:48:28.820440 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 22:48:28.849583 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:48:28.849553 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 22:48:28.858300 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:48:28.858279 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 22:48:36.641299 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:48:36.641229 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-4r6s4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:48:36.642139 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:48:36.642076 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-4r6s4" podUID="d9f99d96-f28e-4152-8285-c6aefa58a412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:48:51.001852 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:48:51.001818 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:58.470380 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:48:58.470266 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:49:28.469275 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:49:28.469234 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:49:47.114186 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:49:47.114107 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:51.320656 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:49:51.320621 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:58.468415 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:49:58.468375 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:49:59.695687 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:49:59.695644 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:50:07.775170 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:50:07.775108 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:50:28.468276 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:50:28.468240 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:50:51.706545 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:50:51.706508 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:50:58.468717 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:50:58.468670 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:51:28.468055 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:51:28.468016 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:51:52.018455 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:51:52.018416 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:51:58.469015 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:51:58.468968 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:52:28.467907 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:52:28.467865 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:52:41.655366 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:52:41.655289 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:41.863255 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:52:41.863222 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:48.285690 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:52:48.285655 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:48.823952 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:52:48.823899 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:49.089683 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:52:49.089592 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:52.315481 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:52:52.315440 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:52.958589 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:52:52.958552 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:53.169459 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:52:53.169426 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:56.074720 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:52:56.074682 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:56.283915 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:52:56.283836 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:58.469869 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:52:58.469828 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:52:59.523882 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:52:59.523835 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:59.726293 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:52:59.726259 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:59.925564 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:52:59.925485 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:04.582632 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:04.582589 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:09.509597 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:09.509555 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:13.334220 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:13.334181 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:14.388534 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:14.388498 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:17.710592 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:17.710559 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:18.262950 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:18.262914 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:18.485587 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:18.485537 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:19.585125 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:19.585093 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:20.775014 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:20.774979 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs"] Nov 02 22:53:20.775423 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:20.775376 4935 cpu_manager.go:414] "containerMap: removing container" podUID="dbc1708b-8a37-4594-a73a-33edbe600fa2" containerName="main" Nov 02 22:53:20.775423 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:20.775387 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbc1708b-8a37-4594-a73a-33edbe600fa2" containerName="main" Nov 02 22:53:20.775504 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:20.775467 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbc1708b-8a37-4594-a73a-33edbe600fa2" containerName="main" Nov 02 22:53:20.778596 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:20.778574 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" Nov 02 22:53:20.828438 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:20.828408 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-90aa9a\"/\"kube-root-ca.crt\"" Nov 02 22:53:20.874388 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:20.874356 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-90aa9a\"/\"simulator-single-main-90aa9a-kserve-self-signed-certs\"" Nov 02 22:53:20.874388 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:20.874383 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-90aa9a\"/\"default-dockercfg-gg45q\"" Nov 02 22:53:20.874614 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:20.874353 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-90aa9a\"/\"openshift-service-ca.crt\"" Nov 02 22:53:20.876492 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:20.876470 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs"] Nov 02 22:53:20.944234 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:20.944178 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:20.964263 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:20.964222 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp57h\" (UniqueName: \"kubernetes.io/projected/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-kube-api-access-lp57h\") pod \"simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs\" (UID: \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\") " pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" Nov 02 22:53:20.964263 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:20.964258 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-tmp-dir\") pod \"simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs\" (UID: \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\") " pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" Nov 02 22:53:20.964475 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:20.964277 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-home\") pod \"simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs\" (UID: \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\") " pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" Nov 02 22:53:20.964475 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:20.964398 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-dshm\") pod \"simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs\" (UID: \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\") " pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" Nov 02 22:53:20.964475 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:20.964442 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-model-cache\") pod \"simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs\" (UID: \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\") " pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" Nov 02 22:53:20.964610 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:20.964505 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-tls-certs\") pod \"simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs\" (UID: \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\") " pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" Nov 02 22:53:21.065380 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:21.065285 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-model-cache\") pod \"simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs\" (UID: \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\") " pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" Nov 02 22:53:21.065380 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:21.065350 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-tls-certs\") pod \"simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs\" (UID: \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\") " pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" Nov 02 22:53:21.065618 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:21.065381 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp57h\" (UniqueName: \"kubernetes.io/projected/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-kube-api-access-lp57h\") pod \"simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs\" (UID: \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\") " pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" Nov 02 22:53:21.065618 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:21.065402 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-tmp-dir\") pod \"simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs\" (UID: \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\") " pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" Nov 02 22:53:21.065618 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:21.065417 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-home\") pod \"simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs\" (UID: \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\") " pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" Nov 02 22:53:21.065618 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:21.065437 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-dshm\") pod \"simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs\" (UID: \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\") " pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" Nov 02 22:53:21.065862 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:21.065766 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-model-cache\") pod \"simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs\" (UID: \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\") " pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" Nov 02 22:53:21.065862 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:21.065781 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-tmp-dir\") pod \"simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs\" (UID: \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\") " pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" Nov 02 22:53:21.065965 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:21.065874 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-home\") pod \"simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs\" (UID: \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\") " pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" Nov 02 22:53:21.067838 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:21.067809 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-dshm\") pod \"simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs\" (UID: \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\") " pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" Nov 02 22:53:21.068123 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:21.068101 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-tls-certs\") pod \"simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs\" (UID: \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\") " pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" Nov 02 22:53:21.118704 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:21.118669 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp57h\" (UniqueName: \"kubernetes.io/projected/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-kube-api-access-lp57h\") pod \"simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs\" (UID: \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\") " pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" Nov 02 22:53:21.399706 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:21.399609 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" Nov 02 22:53:21.766467 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:21.766426 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs"] Nov 02 22:53:21.767657 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:53:21.767628 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17cbf19f_f795_4e92_a540_a57dfa2d2a1a.slice/crio-368db8b7dcdc09d9ab16725801c17554678d74c2666dd1ce9932b6aad624e6a3 WatchSource:0}: Error finding container 368db8b7dcdc09d9ab16725801c17554678d74c2666dd1ce9932b6aad624e6a3: Status 404 returned error can't find the container with id 368db8b7dcdc09d9ab16725801c17554678d74c2666dd1ce9932b6aad624e6a3 Nov 02 22:53:22.640289 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:22.640249 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" event={"ID":"17cbf19f-f795-4e92-a540-a57dfa2d2a1a","Type":"ContainerStarted","Data":"b012e4d4aa50f81fec396494c4a6839c06ff1f4cb6e98997cb8c991740123db3"} Nov 02 22:53:22.640712 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:22.640297 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" event={"ID":"17cbf19f-f795-4e92-a540-a57dfa2d2a1a","Type":"ContainerStarted","Data":"368db8b7dcdc09d9ab16725801c17554678d74c2666dd1ce9932b6aad624e6a3"} Nov 02 22:53:22.747851 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:22.747800 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" podStartSLOduration=2.7477839619999997 podStartE2EDuration="2.747783962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:22.745613392 +0000 UTC m=+6295.185188117" watchObservedRunningTime="2026-11-02 22:53:22.747783962 +0000 UTC m=+6295.187358683" Nov 02 22:53:23.400288 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:23.400248 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" Nov 02 22:53:24.414943 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:24.414905 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:28.469146 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:53:28.469094 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:53:28.853230 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:28.853199 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 22:53:28.863282 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:28.863255 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 22:53:28.889453 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:28.889424 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 22:53:28.899328 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:28.899301 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 22:53:29.274978 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:29.274941 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:31.399757 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:31.399721 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" Nov 02 22:53:31.412299 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:31.412273 4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" Nov 02 22:53:31.700333 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:31.700254 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" Nov 02 22:53:34.173839 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:34.173808 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:39.079924 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:39.079885 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:43.882051 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:43.882018 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:48.787845 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:48.787813 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:52.658555 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:52.658509 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:53.834443 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:53.834408 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:58.469055 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:53:58.469018 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:53:58.620284 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:53:58.620234 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:03.401883 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:03.401846 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:08.300433 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:08.300317 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:13.102991 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:13.102935 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:18.000084 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:18.000044 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:19.338627 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:19.338583 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:19.776730 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:19.776670 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:22.891673 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:22.891637 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:24.223406 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:24.223372 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs"] Nov 02 22:54:24.368892 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:24.368854 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:25.496432 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:25.496394 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:26.494370 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:26.494335 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:26.703098 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:26.703062 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:27.473716 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:27.473679 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:28.470243 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:54:28.470208 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:54:31.039139 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:31.039104 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:36.571475 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:36.571436 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:36.782301 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:36.782262 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:39.243382 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:39.243332 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" podUID="17cbf19f-f795-4e92-a540-a57dfa2d2a1a" containerName="main" containerID="cri-o://b012e4d4aa50f81fec396494c4a6839c06ff1f4cb6e98997cb8c991740123db3" gracePeriod=45 Nov 02 22:54:39.493962 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:39.493897 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" Nov 02 22:54:39.590751 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:39.590723 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-tmp-dir\") pod \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\" (UID: \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\") " Nov 02 22:54:39.590927 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:39.590762 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-kube-api-access-lp57h\" (UniqueName: \"kubernetes.io/projected/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-kube-api-access-lp57h\") pod \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\" (UID: \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\") " Nov 02 22:54:39.590927 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:39.590802 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-tls-certs\" (UniqueName: \"kubernetes.io/secret/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-tls-certs\") pod \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\" (UID: \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\") " Nov 02 22:54:39.590927 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:39.590872 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-model-cache\") pod \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\" (UID: \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\") " Nov 02 22:54:39.591118 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:39.590975 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-home\" (UniqueName: \"kubernetes.io/empty-dir/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-home\") pod \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\" (UID: \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\") " Nov 02 22:54:39.591118 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:39.591007 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-dshm\") pod \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\" (UID: \"17cbf19f-f795-4e92-a540-a57dfa2d2a1a\") " Nov 02 22:54:39.591118 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:39.591049 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-tmp-dir" pod "17cbf19f-f795-4e92-a540-a57dfa2d2a1a" (UID: "17cbf19f-f795-4e92-a540-a57dfa2d2a1a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:54:39.591325 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:39.591203 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-model-cache" pod "17cbf19f-f795-4e92-a540-a57dfa2d2a1a" (UID: "17cbf19f-f795-4e92-a540-a57dfa2d2a1a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:54:39.591325 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:39.591301 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-home" pod "17cbf19f-f795-4e92-a540-a57dfa2d2a1a" (UID: "17cbf19f-f795-4e92-a540-a57dfa2d2a1a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:54:39.591440 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:39.591429 4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-home\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:39.591497 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:39.591447 4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-tmp-dir\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:39.591497 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:39.591462 4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-model-cache\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:39.593074 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:39.593047 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-kube-api-access-lp57h" pod "17cbf19f-f795-4e92-a540-a57dfa2d2a1a" (UID: "17cbf19f-f795-4e92-a540-a57dfa2d2a1a"). InnerVolumeSpecName "kube-api-access-lp57h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:54:39.593398 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:39.593376 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-tls-certs" pod "17cbf19f-f795-4e92-a540-a57dfa2d2a1a" (UID: "17cbf19f-f795-4e92-a540-a57dfa2d2a1a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:54:39.593469 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:39.593394 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-dshm" pod "17cbf19f-f795-4e92-a540-a57dfa2d2a1a" (UID: "17cbf19f-f795-4e92-a540-a57dfa2d2a1a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:54:39.692761 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:39.692723 4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-tls-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:39.692761 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:39.692760 4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-dshm\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:39.692972 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:39.692775 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp57h\" (UniqueName: \"kubernetes.io/projected/17cbf19f-f795-4e92-a540-a57dfa2d2a1a-kube-api-access-lp57h\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:54:40.041442 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:40.041355 4935 generic.go:356] "Generic (PLEG): container finished" podID="17cbf19f-f795-4e92-a540-a57dfa2d2a1a" containerID="b012e4d4aa50f81fec396494c4a6839c06ff1f4cb6e98997cb8c991740123db3" exitCode=0 Nov 02 22:54:40.041674 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:40.041411 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" Nov 02 22:54:40.041796 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:40.041428 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" event={"ID":"17cbf19f-f795-4e92-a540-a57dfa2d2a1a","Type":"ContainerDied","Data":"b012e4d4aa50f81fec396494c4a6839c06ff1f4cb6e98997cb8c991740123db3"} Nov 02 22:54:40.041796 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:40.041768 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs" event={"ID":"17cbf19f-f795-4e92-a540-a57dfa2d2a1a","Type":"ContainerDied","Data":"368db8b7dcdc09d9ab16725801c17554678d74c2666dd1ce9932b6aad624e6a3"} Nov 02 22:54:40.041796 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:40.041783 4935 scope.go:122] "RemoveContainer" containerID="b012e4d4aa50f81fec396494c4a6839c06ff1f4cb6e98997cb8c991740123db3" Nov 02 22:54:40.050573 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:40.050555 4935 scope.go:122] "RemoveContainer" containerID="b012e4d4aa50f81fec396494c4a6839c06ff1f4cb6e98997cb8c991740123db3" Nov 02 22:54:40.050843 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:54:40.050824 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b012e4d4aa50f81fec396494c4a6839c06ff1f4cb6e98997cb8c991740123db3\": container with ID starting with b012e4d4aa50f81fec396494c4a6839c06ff1f4cb6e98997cb8c991740123db3 not found: ID does not exist" containerID="b012e4d4aa50f81fec396494c4a6839c06ff1f4cb6e98997cb8c991740123db3" Nov 02 22:54:40.050901 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:40.050855 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b012e4d4aa50f81fec396494c4a6839c06ff1f4cb6e98997cb8c991740123db3"} err="failed to get container status \"b012e4d4aa50f81fec396494c4a6839c06ff1f4cb6e98997cb8c991740123db3\": rpc error: code = NotFound desc = could not find container \"b012e4d4aa50f81fec396494c4a6839c06ff1f4cb6e98997cb8c991740123db3\": container with ID starting with b012e4d4aa50f81fec396494c4a6839c06ff1f4cb6e98997cb8c991740123db3 not found: ID does not exist" Nov 02 22:54:40.193911 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:40.193880 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs"] Nov 02 22:54:40.239851 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:40.239820 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-90aa9a/simulator-single-main-90aa9a-kserve-7598f79bbc-htzbs"] Nov 02 22:54:41.775070 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:41.775034 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17cbf19f-f795-4e92-a540-a57dfa2d2a1a" path="/var/lib/kubelet/pods/17cbf19f-f795-4e92-a540-a57dfa2d2a1a/volumes" Nov 02 22:54:53.054887 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:54:53.054843 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:58.469473 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:54:58.469427 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:55:01.024963 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:55:01.024929 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:01.509670 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:55:01.509564 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:03.987425 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:55:03.987383 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:04.569183 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:55:04.569151 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:07.340912 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:55:07.340860 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:07.556714 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:55:07.556680 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:24.739901 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:55:24.739868 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xbm86"] Nov 02 22:55:24.740406 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:55:24.740040 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xbm86" podUID="88838f5d-567a-4e5e-9dc9-eff4470e53ff" containerName="manager" containerID="cri-o://29ae623bd265182e771ecdb3fa329e750c8c19460a94d965edd5e5987dd488b5" gracePeriod=10 Nov 02 22:55:26.296514 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:55:26.296478 4935 generic.go:356] "Generic (PLEG): container finished" podID="88838f5d-567a-4e5e-9dc9-eff4470e53ff" containerID="29ae623bd265182e771ecdb3fa329e750c8c19460a94d965edd5e5987dd488b5" exitCode=0 Nov 02 22:55:26.296878 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:55:26.296547 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xbm86" event={"ID":"88838f5d-567a-4e5e-9dc9-eff4470e53ff","Type":"ContainerDied","Data":"29ae623bd265182e771ecdb3fa329e750c8c19460a94d965edd5e5987dd488b5"} Nov 02 22:55:26.403839 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:55:26.403815 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xbm86" Nov 02 22:55:26.536184 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:55:26.536156 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88838f5d-567a-4e5e-9dc9-eff4470e53ff-kube-api-access-ctbgb\" (UniqueName: \"kubernetes.io/projected/88838f5d-567a-4e5e-9dc9-eff4470e53ff-kube-api-access-ctbgb\") pod \"88838f5d-567a-4e5e-9dc9-eff4470e53ff\" (UID: \"88838f5d-567a-4e5e-9dc9-eff4470e53ff\") " Nov 02 22:55:26.536350 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:55:26.536291 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88838f5d-567a-4e5e-9dc9-eff4470e53ff-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88838f5d-567a-4e5e-9dc9-eff4470e53ff-extensions-socket-volume\") pod \"88838f5d-567a-4e5e-9dc9-eff4470e53ff\" (UID: \"88838f5d-567a-4e5e-9dc9-eff4470e53ff\") " Nov 02 22:55:26.536661 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:55:26.536640 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88838f5d-567a-4e5e-9dc9-eff4470e53ff-extensions-socket-volume" pod "88838f5d-567a-4e5e-9dc9-eff4470e53ff" (UID: "88838f5d-567a-4e5e-9dc9-eff4470e53ff"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:55:26.538109 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:55:26.538083 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88838f5d-567a-4e5e-9dc9-eff4470e53ff-kube-api-access-ctbgb" pod "88838f5d-567a-4e5e-9dc9-eff4470e53ff" (UID: "88838f5d-567a-4e5e-9dc9-eff4470e53ff"). InnerVolumeSpecName "kube-api-access-ctbgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:55:26.637597 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:55:26.637564 4935 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88838f5d-567a-4e5e-9dc9-eff4470e53ff-extensions-socket-volume\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:55:26.637597 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:55:26.637594 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctbgb\" (UniqueName: \"kubernetes.io/projected/88838f5d-567a-4e5e-9dc9-eff4470e53ff-kube-api-access-ctbgb\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:55:27.302559 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:55:27.302525 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xbm86" Nov 02 22:55:27.302559 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:55:27.302551 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xbm86" event={"ID":"88838f5d-567a-4e5e-9dc9-eff4470e53ff","Type":"ContainerDied","Data":"dfc1ec9750966c1939e5a3c00d24eec3af9a55f089e85e622f0f2604081ad857"} Nov 02 22:55:27.303068 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:55:27.302618 4935 scope.go:122] "RemoveContainer" containerID="29ae623bd265182e771ecdb3fa329e750c8c19460a94d965edd5e5987dd488b5" Nov 02 22:55:27.501811 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:55:27.501778 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xbm86"] Nov 02 22:55:27.548268 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:55:27.548234 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xbm86"] Nov 02 22:55:27.776902 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:55:27.776866 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88838f5d-567a-4e5e-9dc9-eff4470e53ff" path="/var/lib/kubelet/pods/88838f5d-567a-4e5e-9dc9-eff4470e53ff/volumes" Nov 02 22:55:28.468238 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:55:28.468202 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:55:42.174851 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:55:42.174765 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:42.374712 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:55:42.374662 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:53.345602 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:55:53.345552 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:58.468657 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:55:58.468619 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:56:25.068239 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:56:25.068199 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:25.289786 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:56:25.289750 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:28.327732 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:56:28.327693 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8xz8j"] Nov 02 22:56:28.328180 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:56:28.328080 4935 cpu_manager.go:414] "containerMap: removing container" podUID="17cbf19f-f795-4e92-a540-a57dfa2d2a1a" containerName="main" Nov 02 22:56:28.328180 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:56:28.328091 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17cbf19f-f795-4e92-a540-a57dfa2d2a1a" containerName="main" Nov 02 22:56:28.328180 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:56:28.328144 4935 cpu_manager.go:414] "containerMap: removing container" podUID="88838f5d-567a-4e5e-9dc9-eff4470e53ff" containerName="manager" Nov 02 22:56:28.328180 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:56:28.328149 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88838f5d-567a-4e5e-9dc9-eff4470e53ff" containerName="manager" Nov 02 22:56:28.328345 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:56:28.328205 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="88838f5d-567a-4e5e-9dc9-eff4470e53ff" containerName="manager" Nov 02 22:56:28.328345 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:56:28.328213 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="17cbf19f-f795-4e92-a540-a57dfa2d2a1a" containerName="main" Nov 02 22:56:28.331159 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:56:28.331119 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8xz8j" Nov 02 22:56:28.378606 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:56:28.378577 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8xz8j"] Nov 02 22:56:28.422609 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:56:28.422584 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-z8k9l\"" Nov 02 22:56:28.468904 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:56:28.468867 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:56:28.469067 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:56:28.469046 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/808ca2fc-b968-4d50-9276-273bedb83a22-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8xz8j\" (UID: \"808ca2fc-b968-4d50-9276-273bedb83a22\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8xz8j" Nov 02 22:56:28.469155 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:56:28.469120 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmbjh\" (UniqueName: \"kubernetes.io/projected/808ca2fc-b968-4d50-9276-273bedb83a22-kube-api-access-vmbjh\") pod \"kuadrant-operator-controller-manager-678d87766f-8xz8j\" (UID: \"808ca2fc-b968-4d50-9276-273bedb83a22\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8xz8j" Nov 02 22:56:28.570451 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:56:28.570413 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/808ca2fc-b968-4d50-9276-273bedb83a22-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8xz8j\" (UID: \"808ca2fc-b968-4d50-9276-273bedb83a22\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8xz8j" Nov 02 22:56:28.570665 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:56:28.570577 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmbjh\" (UniqueName: \"kubernetes.io/projected/808ca2fc-b968-4d50-9276-273bedb83a22-kube-api-access-vmbjh\") pod \"kuadrant-operator-controller-manager-678d87766f-8xz8j\" (UID: \"808ca2fc-b968-4d50-9276-273bedb83a22\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8xz8j" Nov 02 22:56:28.570879 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:56:28.570856 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/808ca2fc-b968-4d50-9276-273bedb83a22-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8xz8j\" (UID: \"808ca2fc-b968-4d50-9276-273bedb83a22\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8xz8j" Nov 02 22:56:28.625768 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:56:28.625695 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmbjh\" (UniqueName: \"kubernetes.io/projected/808ca2fc-b968-4d50-9276-273bedb83a22-kube-api-access-vmbjh\") pod \"kuadrant-operator-controller-manager-678d87766f-8xz8j\" (UID: \"808ca2fc-b968-4d50-9276-273bedb83a22\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8xz8j" Nov 02 22:56:28.643521 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:56:28.643494 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8xz8j" Nov 02 22:56:29.108518 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:56:29.108489 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8xz8j"] Nov 02 22:56:29.111005 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:56:29.110962 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod808ca2fc_b968_4d50_9276_273bedb83a22.slice/crio-445377160f15ed85c41bbb0d4ad5d6aa5fd85fba06acf78e3aa9f4e231d07b37 WatchSource:0}: Error finding container 445377160f15ed85c41bbb0d4ad5d6aa5fd85fba06acf78e3aa9f4e231d07b37: Status 404 returned error can't find the container with id 445377160f15ed85c41bbb0d4ad5d6aa5fd85fba06acf78e3aa9f4e231d07b37 Nov 02 22:56:29.613540 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:56:29.613504 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8xz8j" event={"ID":"808ca2fc-b968-4d50-9276-273bedb83a22","Type":"ContainerStarted","Data":"08d049718c903db89804de8742793579ee5b82357eec3bd39ca1acac01f5f621"} Nov 02 22:56:29.614013 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:56:29.613551 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8xz8j" event={"ID":"808ca2fc-b968-4d50-9276-273bedb83a22","Type":"ContainerStarted","Data":"445377160f15ed85c41bbb0d4ad5d6aa5fd85fba06acf78e3aa9f4e231d07b37"} Nov 02 22:56:29.614013 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:56:29.613592 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8xz8j" Nov 02 22:56:29.721014 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:56:29.720966 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8xz8j" podStartSLOduration=1.720952679 podStartE2EDuration="1.720952679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:56:29.72063488 +0000 UTC m=+6482.160209602" watchObservedRunningTime="2026-11-02 22:56:29.720952679 +0000 UTC m=+6482.160527395" Nov 02 22:56:40.620277 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:56:40.620245 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8xz8j" Nov 02 22:56:53.684702 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:56:53.684669 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:58.468504 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:56:58.468461 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:57:28.468862 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:57:28.468823 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:57:37.389046 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:37.389002 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7c9bd99f4f-dp79r"] Nov 02 22:57:37.392696 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:37.392672 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-dp79r" Nov 02 22:57:37.464376 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:37.464334 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8rf4\" (UniqueName: \"kubernetes.io/projected/8ad0482a-fdd3-42ff-9203-8040bbc545b6-kube-api-access-z8rf4\") pod \"maas-api-7c9bd99f4f-dp79r\" (UID: \"8ad0482a-fdd3-42ff-9203-8040bbc545b6\") " pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-dp79r" Nov 02 22:57:37.464556 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:37.464397 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ad0482a-fdd3-42ff-9203-8040bbc545b6-metrics-certs\") pod \"maas-api-7c9bd99f4f-dp79r\" (UID: \"8ad0482a-fdd3-42ff-9203-8040bbc545b6\") " pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-dp79r" Nov 02 22:57:37.464556 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:37.464473 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ad0482a-fdd3-42ff-9203-8040bbc545b6-maas-api-tls\") pod \"maas-api-7c9bd99f4f-dp79r\" (UID: \"8ad0482a-fdd3-42ff-9203-8040bbc545b6\") " pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-dp79r" Nov 02 22:57:37.464556 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:37.464544 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ad0482a-fdd3-42ff-9203-8040bbc545b6-openshift-service-ca\") pod \"maas-api-7c9bd99f4f-dp79r\" (UID: \"8ad0482a-fdd3-42ff-9203-8040bbc545b6\") " pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-dp79r" Nov 02 22:57:37.490897 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:37.490855 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c9bd99f4f-dp79r"] Nov 02 22:57:37.565938 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:37.565897 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ad0482a-fdd3-42ff-9203-8040bbc545b6-maas-api-tls\") pod \"maas-api-7c9bd99f4f-dp79r\" (UID: \"8ad0482a-fdd3-42ff-9203-8040bbc545b6\") " pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-dp79r" Nov 02 22:57:37.566173 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:37.565957 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ad0482a-fdd3-42ff-9203-8040bbc545b6-openshift-service-ca\") pod \"maas-api-7c9bd99f4f-dp79r\" (UID: \"8ad0482a-fdd3-42ff-9203-8040bbc545b6\") " pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-dp79r" Nov 02 22:57:37.566173 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:37.566066 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8rf4\" (UniqueName: \"kubernetes.io/projected/8ad0482a-fdd3-42ff-9203-8040bbc545b6-kube-api-access-z8rf4\") pod \"maas-api-7c9bd99f4f-dp79r\" (UID: \"8ad0482a-fdd3-42ff-9203-8040bbc545b6\") " pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-dp79r" Nov 02 22:57:37.566173 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:37.566159 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ad0482a-fdd3-42ff-9203-8040bbc545b6-metrics-certs\") pod \"maas-api-7c9bd99f4f-dp79r\" (UID: \"8ad0482a-fdd3-42ff-9203-8040bbc545b6\") " pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-dp79r" Nov 02 22:57:37.566698 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:37.566672 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ad0482a-fdd3-42ff-9203-8040bbc545b6-openshift-service-ca\") pod \"maas-api-7c9bd99f4f-dp79r\" (UID: \"8ad0482a-fdd3-42ff-9203-8040bbc545b6\") " pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-dp79r" Nov 02 22:57:37.568602 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:37.568576 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ad0482a-fdd3-42ff-9203-8040bbc545b6-metrics-certs\") pod \"maas-api-7c9bd99f4f-dp79r\" (UID: \"8ad0482a-fdd3-42ff-9203-8040bbc545b6\") " pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-dp79r" Nov 02 22:57:37.568732 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:37.568712 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ad0482a-fdd3-42ff-9203-8040bbc545b6-maas-api-tls\") pod \"maas-api-7c9bd99f4f-dp79r\" (UID: \"8ad0482a-fdd3-42ff-9203-8040bbc545b6\") " pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-dp79r" Nov 02 22:57:37.618985 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:37.618953 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8rf4\" (UniqueName: \"kubernetes.io/projected/8ad0482a-fdd3-42ff-9203-8040bbc545b6-kube-api-access-z8rf4\") pod \"maas-api-7c9bd99f4f-dp79r\" (UID: \"8ad0482a-fdd3-42ff-9203-8040bbc545b6\") " pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-dp79r" Nov 02 22:57:37.705780 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:37.705697 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-dp79r" Nov 02 22:57:38.072955 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:38.072927 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c9bd99f4f-dp79r"] Nov 02 22:57:38.074864 ip-10-0-128-89 kubenswrapper[4935]: W1102 22:57:38.074829 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad0482a_fdd3_42ff_9203_8040bbc545b6.slice/crio-9cf6266f99d447d9778b293715973ad3ad6caf2838b6eb901091bead56f5df11 WatchSource:0}: Error finding container 9cf6266f99d447d9778b293715973ad3ad6caf2838b6eb901091bead56f5df11: Status 404 returned error can't find the container with id 9cf6266f99d447d9778b293715973ad3ad6caf2838b6eb901091bead56f5df11 Nov 02 22:57:38.076170 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:38.076153 4935 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:57:39.080911 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:39.080866 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-dp79r" event={"ID":"8ad0482a-fdd3-42ff-9203-8040bbc545b6","Type":"ContainerStarted","Data":"3062077b5ea4f202618717709201c78b9f1e20031f5269d79bb8534424d5a48a"} Nov 02 22:57:39.081324 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:39.080939 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-dp79r" Nov 02 22:57:39.081324 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:39.080966 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-dp79r" event={"ID":"8ad0482a-fdd3-42ff-9203-8040bbc545b6","Type":"ContainerStarted","Data":"9cf6266f99d447d9778b293715973ad3ad6caf2838b6eb901091bead56f5df11"} Nov 02 22:57:39.231294 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:39.231185 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-dp79r" podStartSLOduration=1.334347672 podStartE2EDuration="2.231169999s" totalImagesPullingTime="896.822327ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:57:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:57:39.229340827 +0000 UTC m=+6551.668915549" watchObservedRunningTime="2026-11-02 22:57:39.231169999 +0000 UTC m=+6551.670744722" Nov 02 22:57:45.091581 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:45.091546 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7c9bd99f4f-dp79r" Nov 02 22:57:45.248166 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:45.248108 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-bbf944969-v2c9t"] Nov 02 22:57:45.248410 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:45.248366 4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-bbf944969-v2c9t" podUID="e97812f0-4d82-49eb-94b1-27274f2c0f90" containerName="maas-api" containerID="cri-o://e5208b9d6fa7bc161be9cdff608571eeb1fab4077c1fece3872bf6ff1d4719f4" gracePeriod=30 Nov 02 22:57:46.505893 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:46.505864 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bbf944969-v2c9t" Nov 02 22:57:46.560206 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:46.560111 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e97812f0-4d82-49eb-94b1-27274f2c0f90-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e97812f0-4d82-49eb-94b1-27274f2c0f90-maas-api-tls\") pod \"e97812f0-4d82-49eb-94b1-27274f2c0f90\" (UID: \"e97812f0-4d82-49eb-94b1-27274f2c0f90\") " Nov 02 22:57:46.560206 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:46.560193 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e97812f0-4d82-49eb-94b1-27274f2c0f90-kube-api-access-fmsmn\" (UniqueName: \"kubernetes.io/projected/e97812f0-4d82-49eb-94b1-27274f2c0f90-kube-api-access-fmsmn\") pod \"e97812f0-4d82-49eb-94b1-27274f2c0f90\" (UID: \"e97812f0-4d82-49eb-94b1-27274f2c0f90\") " Nov 02 22:57:46.560422 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:46.560301 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e97812f0-4d82-49eb-94b1-27274f2c0f90-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e97812f0-4d82-49eb-94b1-27274f2c0f90-metrics-certs\") pod \"e97812f0-4d82-49eb-94b1-27274f2c0f90\" (UID: \"e97812f0-4d82-49eb-94b1-27274f2c0f90\") " Nov 02 22:57:46.560422 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:46.560325 4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e97812f0-4d82-49eb-94b1-27274f2c0f90-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e97812f0-4d82-49eb-94b1-27274f2c0f90-openshift-service-ca\") pod \"e97812f0-4d82-49eb-94b1-27274f2c0f90\" (UID: \"e97812f0-4d82-49eb-94b1-27274f2c0f90\") " Nov 02 22:57:46.560789 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:46.560760 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e97812f0-4d82-49eb-94b1-27274f2c0f90-openshift-service-ca" pod "e97812f0-4d82-49eb-94b1-27274f2c0f90" (UID: "e97812f0-4d82-49eb-94b1-27274f2c0f90"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:57:46.562321 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:46.562297 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e97812f0-4d82-49eb-94b1-27274f2c0f90-kube-api-access-fmsmn" pod "e97812f0-4d82-49eb-94b1-27274f2c0f90" (UID: "e97812f0-4d82-49eb-94b1-27274f2c0f90"). InnerVolumeSpecName "kube-api-access-fmsmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:57:46.562744 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:46.562721 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e97812f0-4d82-49eb-94b1-27274f2c0f90-metrics-certs" pod "e97812f0-4d82-49eb-94b1-27274f2c0f90" (UID: "e97812f0-4d82-49eb-94b1-27274f2c0f90"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:57:46.562744 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:46.562737 4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e97812f0-4d82-49eb-94b1-27274f2c0f90-maas-api-tls" pod "e97812f0-4d82-49eb-94b1-27274f2c0f90" (UID: "e97812f0-4d82-49eb-94b1-27274f2c0f90"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:57:46.661467 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:46.661427 4935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e97812f0-4d82-49eb-94b1-27274f2c0f90-openshift-service-ca\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:57:46.661467 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:46.661461 4935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e97812f0-4d82-49eb-94b1-27274f2c0f90-maas-api-tls\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:57:46.661467 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:46.661471 4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmsmn\" (UniqueName: \"kubernetes.io/projected/e97812f0-4d82-49eb-94b1-27274f2c0f90-kube-api-access-fmsmn\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:57:46.661719 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:46.661482 4935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e97812f0-4d82-49eb-94b1-27274f2c0f90-metrics-certs\") on node \"ip-10-0-128-89.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:57:47.134716 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:47.134678 4935 generic.go:356] "Generic (PLEG): container finished" podID="e97812f0-4d82-49eb-94b1-27274f2c0f90" containerID="e5208b9d6fa7bc161be9cdff608571eeb1fab4077c1fece3872bf6ff1d4719f4" exitCode=0 Nov 02 22:57:47.134934 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:47.134732 4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bbf944969-v2c9t" Nov 02 22:57:47.134934 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:47.134740 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bbf944969-v2c9t" event={"ID":"e97812f0-4d82-49eb-94b1-27274f2c0f90","Type":"ContainerDied","Data":"e5208b9d6fa7bc161be9cdff608571eeb1fab4077c1fece3872bf6ff1d4719f4"} Nov 02 22:57:47.134934 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:47.134846 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bbf944969-v2c9t" event={"ID":"e97812f0-4d82-49eb-94b1-27274f2c0f90","Type":"ContainerDied","Data":"d427c3a0add04d2cef27877966dc7a8ba2db60ecf587e10b9a31a06cf1f02395"} Nov 02 22:57:47.134934 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:47.134870 4935 scope.go:122] "RemoveContainer" containerID="e5208b9d6fa7bc161be9cdff608571eeb1fab4077c1fece3872bf6ff1d4719f4" Nov 02 22:57:47.145715 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:47.145692 4935 scope.go:122] "RemoveContainer" containerID="e5208b9d6fa7bc161be9cdff608571eeb1fab4077c1fece3872bf6ff1d4719f4" Nov 02 22:57:47.145970 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:57:47.145954 4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5208b9d6fa7bc161be9cdff608571eeb1fab4077c1fece3872bf6ff1d4719f4\": container with ID starting with e5208b9d6fa7bc161be9cdff608571eeb1fab4077c1fece3872bf6ff1d4719f4 not found: ID does not exist" containerID="e5208b9d6fa7bc161be9cdff608571eeb1fab4077c1fece3872bf6ff1d4719f4" Nov 02 22:57:47.146018 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:47.145979 4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5208b9d6fa7bc161be9cdff608571eeb1fab4077c1fece3872bf6ff1d4719f4"} err="failed to get container status \"e5208b9d6fa7bc161be9cdff608571eeb1fab4077c1fece3872bf6ff1d4719f4\": rpc error: code = NotFound desc = could not find container \"e5208b9d6fa7bc161be9cdff608571eeb1fab4077c1fece3872bf6ff1d4719f4\": container with ID starting with e5208b9d6fa7bc161be9cdff608571eeb1fab4077c1fece3872bf6ff1d4719f4 not found: ID does not exist" Nov 02 22:57:47.290908 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:47.290865 4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-bbf944969-v2c9t"] Nov 02 22:57:47.337025 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:47.336991 4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-bbf944969-v2c9t"] Nov 02 22:57:47.775325 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:47.775279 4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e97812f0-4d82-49eb-94b1-27274f2c0f90" path="/var/lib/kubelet/pods/e97812f0-4d82-49eb-94b1-27274f2c0f90/volumes" Nov 02 22:57:54.018989 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:57:54.018941 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:57:58.468051 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:57:58.468013 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:58:28.469577 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:58:28.469541 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:58:28.891841 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:58:28.891812 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 22:58:28.901330 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:58:28.901306 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 22:58:28.929726 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:58:28.929701 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 22:58:28.938826 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:58:28.938802 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 22:58:54.342024 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:58:54.341934 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:58:58.515014 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:58:58.514971 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:59:22.753331 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:59:22.753298 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:59:22.959355 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:59:22.959317 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:59:28.468111 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:59:28.468071 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 22:59:54.702470 ip-10-0-128-89 kubenswrapper[4935]: I1102 22:59:54.702421 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 22:59:58.470295 ip-10-0-128-89 kubenswrapper[4935]: E1102 22:59:58.470229 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 23:00:07.290298 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:00:07.290253 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:07.522502 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:00:07.522462 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:12.246326 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:00:12.246244 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:12.440885 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:00:12.440852 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:28.468420 ip-10-0-128-89 kubenswrapper[4935]: E1102 23:00:28.468376 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 23:00:54.987468 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:00:54.987383 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:58.468160 ip-10-0-128-89 kubenswrapper[4935]: E1102 23:00:58.468108 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 23:01:28.467761 ip-10-0-128-89 kubenswrapper[4935]: E1102 23:01:28.467712 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 23:01:55.395468 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:01:55.395356 4935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-hpgpb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 23:01:55.395468 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:01:55.395445 4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-hpgpb" podUID="371ba12b-5ead-4acc-aa5d-1de3d1e97b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 23:01:55.405881 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:01:55.405845 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:58.467846 ip-10-0-128-89 kubenswrapper[4935]: E1102 23:01:58.467808 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 23:02:28.468424 ip-10-0-128-89 kubenswrapper[4935]: E1102 23:02:28.468375 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 23:02:49.577037 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:02:49.576995 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:55.723542 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:02:55.723507 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:58.516811 ip-10-0-128-89 kubenswrapper[4935]: E1102 23:02:58.516769 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 23:03:04.683716 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:03:04.683676 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 23:03:28.469076 ip-10-0-128-89 kubenswrapper[4935]: E1102 23:03:28.469026 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 23:03:28.927686 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:03:28.927648 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 23:03:28.936760 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:03:28.936735 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 23:03:28.975413 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:03:28.975384 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-qd9n6_9a2820a3-6d1e-42d0-8a89-0e47f870abc4/console-operator/3.log" Nov 02 23:03:28.984002 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:03:28.983979 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqnmg_bff8d097-26a3-4621-ac01-c3b00ed23e06/ovn-acl-logging/0.log" Nov 02 23:03:55.054098 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:03:55.054063 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-dhqhh_c6839ea0-0dae-40a5-a5c3-94220c3c70de/manager/0.log" Nov 02 23:03:55.403707 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:03:55.403625 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-28jss_cd2913d9-e59b-4b4a-b10a-a324e0c62f25/manager/0.log" Nov 02 23:03:56.051577 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:03:56.051543 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 23:03:56.796608 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:03:56.796573 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-nmwv9_0d2b5b50-c80c-4bac-8830-6c38da5fee20/proxy/0.log" Nov 02 23:03:57.749846 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:03:57.749813 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7c9bd99f4f-dp79r_8ad0482a-fdd3-42ff-9203-8040bbc545b6/maas-api/0.log" Nov 02 23:03:58.468181 ip-10-0-128-89 kubenswrapper[4935]: E1102 23:03:58.468144 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 23:04:01.616005 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:01.615968 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-z2tlv_aa2ea710-8501-46f7-bb15-bca0cf39cb6f/postgres/0.log" Nov 02 23:04:03.802748 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:03.802720 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-gh5gd_42f901f6-b299-4cc4-8871-2753cb497f94/manager/0.log" Nov 02 23:04:04.551854 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:04.551807 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-8xz8j_808ca2fc-b968-4d50-9276-273bedb83a22/manager/0.log" Nov 02 23:04:05.291847 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:05.291815 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-8ggc5_7797919e-d037-4b9c-9eb9-3d0d8af052fb/manager/0.log" Nov 02 23:04:06.229481 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:06.229424 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7c689579c-cvz5g_d52ce2cb-6893-4347-b3aa-6268aae6ceda/istio-proxy/0.log" Nov 02 23:04:07.167032 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:07.167000 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-4r6s4_d9f99d96-f28e-4152-8285-c6aefa58a412/istio-proxy/1.log" Nov 02 23:04:07.713338 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:07.713311 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-6fb52_b78539b8-fd5e-4fd6-a00d-95857df77139/istio-proxy/0.log" Nov 02 23:04:08.445912 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:08.445881 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-hpgpb_371ba12b-5ead-4acc-aa5d-1de3d1e97b64/istio-proxy/0.log" Nov 02 23:04:09.627750 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:09.627715 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr_c8acbb0a-9fb9-494f-97d3-05385082fae6/istio-proxy/0.log" Nov 02 23:04:10.297471 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:10.297441 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-798b7b6b45-hlcl7_44a209b7-2144-45a0-bea0-2f8adf6658cb/router/0.log" Nov 02 23:04:11.067934 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:11.067900 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-4r6s4_d9f99d96-f28e-4152-8285-c6aefa58a412/istio-proxy/0.log" Nov 02 23:04:22.319353 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:22.319315 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hpr2l/must-gather-8q9q9"] Nov 02 23:04:22.321747 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:22.319699 4935 cpu_manager.go:414] "containerMap: removing container" podUID="e97812f0-4d82-49eb-94b1-27274f2c0f90" containerName="maas-api" Nov 02 23:04:22.321747 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:22.319711 4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e97812f0-4d82-49eb-94b1-27274f2c0f90" containerName="maas-api" Nov 02 23:04:22.321747 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:22.319805 4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="e97812f0-4d82-49eb-94b1-27274f2c0f90" containerName="maas-api" Nov 02 23:04:22.322927 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:22.322906 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hpr2l/must-gather-8q9q9" Nov 02 23:04:22.331614 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:22.331591 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 23:04:22.370656 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:22.370629 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjrfr\" (UniqueName: \"kubernetes.io/projected/b0539ee4-eb19-4fb4-9bca-dd967ce54a71-kube-api-access-vjrfr\") pod \"must-gather-8q9q9\" (UID: \"b0539ee4-eb19-4fb4-9bca-dd967ce54a71\") " pod="openshift-must-gather-hpr2l/must-gather-8q9q9" Nov 02 23:04:22.370765 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:22.370661 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b0539ee4-eb19-4fb4-9bca-dd967ce54a71-must-gather-output\") pod \"must-gather-8q9q9\" (UID: \"b0539ee4-eb19-4fb4-9bca-dd967ce54a71\") " pod="openshift-must-gather-hpr2l/must-gather-8q9q9" Nov 02 23:04:22.418840 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:22.418817 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hpr2l\"/\"kube-root-ca.crt\"" Nov 02 23:04:22.418981 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:22.418849 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hpr2l\"/\"openshift-service-ca.crt\"" Nov 02 23:04:22.418981 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:22.418875 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hpr2l\"/\"default-dockercfg-ld5wz\"" Nov 02 23:04:22.420978 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:22.420955 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hpr2l/must-gather-8q9q9"] Nov 02 23:04:22.471657 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:22.471635 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjrfr\" (UniqueName: \"kubernetes.io/projected/b0539ee4-eb19-4fb4-9bca-dd967ce54a71-kube-api-access-vjrfr\") pod \"must-gather-8q9q9\" (UID: \"b0539ee4-eb19-4fb4-9bca-dd967ce54a71\") " pod="openshift-must-gather-hpr2l/must-gather-8q9q9" Nov 02 23:04:22.471770 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:22.471665 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b0539ee4-eb19-4fb4-9bca-dd967ce54a71-must-gather-output\") pod \"must-gather-8q9q9\" (UID: \"b0539ee4-eb19-4fb4-9bca-dd967ce54a71\") " pod="openshift-must-gather-hpr2l/must-gather-8q9q9" Nov 02 23:04:22.471944 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:22.471930 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b0539ee4-eb19-4fb4-9bca-dd967ce54a71-must-gather-output\") pod \"must-gather-8q9q9\" (UID: \"b0539ee4-eb19-4fb4-9bca-dd967ce54a71\") " pod="openshift-must-gather-hpr2l/must-gather-8q9q9" Nov 02 23:04:22.529034 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:22.528990 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjrfr\" (UniqueName: \"kubernetes.io/projected/b0539ee4-eb19-4fb4-9bca-dd967ce54a71-kube-api-access-vjrfr\") pod \"must-gather-8q9q9\" (UID: \"b0539ee4-eb19-4fb4-9bca-dd967ce54a71\") " pod="openshift-must-gather-hpr2l/must-gather-8q9q9" Nov 02 23:04:22.590167 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:22.590078 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 23:04:22.653753 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:22.653724 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hpr2l/must-gather-8q9q9" Nov 02 23:04:23.017556 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:23.017393 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hpr2l/must-gather-8q9q9"] Nov 02 23:04:23.020037 ip-10-0-128-89 kubenswrapper[4935]: W1102 23:04:23.020009 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0539ee4_eb19_4fb4_9bca_dd967ce54a71.slice/crio-56af2f1e47a67e3b2850c37e34b8a2bf73bf04c179080af86a77c31cc9237e1d WatchSource:0}: Error finding container 56af2f1e47a67e3b2850c37e34b8a2bf73bf04c179080af86a77c31cc9237e1d: Status 404 returned error can't find the container with id 56af2f1e47a67e3b2850c37e34b8a2bf73bf04c179080af86a77c31cc9237e1d Nov 02 23:04:23.021978 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:23.021960 4935 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:04:23.539204 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:23.539163 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hpr2l/must-gather-8q9q9" event={"ID":"b0539ee4-eb19-4fb4-9bca-dd967ce54a71","Type":"ContainerStarted","Data":"56af2f1e47a67e3b2850c37e34b8a2bf73bf04c179080af86a77c31cc9237e1d"} Nov 02 23:04:28.474460 ip-10-0-128-89 kubenswrapper[4935]: E1102 23:04:28.474406 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 23:04:30.581775 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:30.581679 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hpr2l/must-gather-8q9q9" event={"ID":"b0539ee4-eb19-4fb4-9bca-dd967ce54a71","Type":"ContainerStarted","Data":"ad8b0ec69a255c45ab42016b40307ba9ce14419d6a2a703585f7345be80e1ffc"} Nov 02 23:04:30.581775 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:30.581725 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hpr2l/must-gather-8q9q9" event={"ID":"b0539ee4-eb19-4fb4-9bca-dd967ce54a71","Type":"ContainerStarted","Data":"4fe0838902d75d673608dc1153836684e14a2ae9f3db2306822fe00600fbb7bd"} Nov 02 23:04:30.686659 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:30.686602 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-hpr2l/must-gather-8q9q9" podStartSLOduration=1.40905562 podStartE2EDuration="8.686587545s" totalImagesPullingTime="7.277531925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:04:30.685395427 +0000 UTC m=+6963.124970149" watchObservedRunningTime="2026-11-02 23:04:30.686587545 +0000 UTC m=+6963.126162266" Nov 02 23:04:31.546323 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:31.546288 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bllxh_87565769-a30f-4ce5-b0d4-0bb6ccf2a4f1/global-pull-secret-syncer/0.log" Nov 02 23:04:33.252616 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:33.252590 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-5zc9z_d3acf985-6085-4061-8ca8-e6d7036cb6fd/connection-checker/0.log" Nov 02 23:04:33.657803 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:33.657771 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-bb7st_c4a07e29-b928-4427-b4c2-5315984f03b3/connection-checker/0.log" Nov 02 23:04:34.106699 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:34.106671 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-nv2sw_345370c7-0ca8-4afe-94db-c9f5394d4d72/connection-checker/0.log" Nov 02 23:04:34.543105 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:34.543064 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" Nov 02 23:04:34.874981 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:34.874909 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-dhqhh_c6839ea0-0dae-40a5-a5c3-94220c3c70de/manager/0.log" Nov 02 23:04:35.567740 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:35.567698 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-28jss_cd2913d9-e59b-4b4a-b10a-a324e0c62f25/manager/0.log" Nov 02 23:04:35.867497 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:35.867423 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-89.us-west-2.compute.internal_398d07e61029ddf8dea61a5829612cad/haproxy/0.log" Nov 02 23:04:36.950144 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:36.950096 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-nmwv9_0d2b5b50-c80c-4bac-8830-6c38da5fee20/proxy/0.log" Nov 02 23:04:40.434325 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:40.434248 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-dhqhh_c6839ea0-0dae-40a5-a5c3-94220c3c70de/manager/0.log" Nov 02 23:04:41.303069 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:41.303035 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-28jss_cd2913d9-e59b-4b4a-b10a-a324e0c62f25/manager/0.log" Nov 02 23:04:41.637686 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:41.637605 4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vlpkd/perf-node-gather-daemonset-lfb9h"] Nov 02 23:04:41.645508 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:41.645483 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-lfb9h" Nov 02 23:04:41.740416 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:41.740383 4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vlpkd\"/\"default-dockercfg-v24v2\"" Nov 02 23:04:41.740416 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:41.740387 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vlpkd\"/\"openshift-service-ca.crt\"" Nov 02 23:04:41.740693 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:41.740433 4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vlpkd\"/\"kube-root-ca.crt\"" Nov 02 23:04:41.742868 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:41.742841 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vlpkd/perf-node-gather-daemonset-lfb9h"] Nov 02 23:04:41.753192 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:41.753170 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a18b612d-eb27-4d6d-a508-5fa818b786ed-podres\") pod \"perf-node-gather-daemonset-lfb9h\" (UID: \"a18b612d-eb27-4d6d-a508-5fa818b786ed\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-lfb9h" Nov 02 23:04:41.753300 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:41.753215 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a18b612d-eb27-4d6d-a508-5fa818b786ed-lib-modules\") pod \"perf-node-gather-daemonset-lfb9h\" (UID: \"a18b612d-eb27-4d6d-a508-5fa818b786ed\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-lfb9h" Nov 02 23:04:41.753300 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:41.753280 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a18b612d-eb27-4d6d-a508-5fa818b786ed-sys\") pod \"perf-node-gather-daemonset-lfb9h\" (UID: \"a18b612d-eb27-4d6d-a508-5fa818b786ed\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-lfb9h" Nov 02 23:04:41.753407 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:41.753312 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57bj7\" (UniqueName: \"kubernetes.io/projected/a18b612d-eb27-4d6d-a508-5fa818b786ed-kube-api-access-57bj7\") pod \"perf-node-gather-daemonset-lfb9h\" (UID: \"a18b612d-eb27-4d6d-a508-5fa818b786ed\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-lfb9h" Nov 02 23:04:41.753407 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:41.753370 4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a18b612d-eb27-4d6d-a508-5fa818b786ed-proc\") pod \"perf-node-gather-daemonset-lfb9h\" (UID: \"a18b612d-eb27-4d6d-a508-5fa818b786ed\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-lfb9h" Nov 02 23:04:41.854232 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:41.854196 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a18b612d-eb27-4d6d-a508-5fa818b786ed-proc\") pod \"perf-node-gather-daemonset-lfb9h\" (UID: \"a18b612d-eb27-4d6d-a508-5fa818b786ed\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-lfb9h" Nov 02 23:04:41.854422 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:41.854275 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a18b612d-eb27-4d6d-a508-5fa818b786ed-podres\") pod \"perf-node-gather-daemonset-lfb9h\" (UID: \"a18b612d-eb27-4d6d-a508-5fa818b786ed\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-lfb9h" Nov 02 23:04:41.854422 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:41.854322 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a18b612d-eb27-4d6d-a508-5fa818b786ed-lib-modules\") pod \"perf-node-gather-daemonset-lfb9h\" (UID: \"a18b612d-eb27-4d6d-a508-5fa818b786ed\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-lfb9h" Nov 02 23:04:41.854422 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:41.854325 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a18b612d-eb27-4d6d-a508-5fa818b786ed-proc\") pod \"perf-node-gather-daemonset-lfb9h\" (UID: \"a18b612d-eb27-4d6d-a508-5fa818b786ed\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-lfb9h" Nov 02 23:04:41.854422 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:41.854349 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a18b612d-eb27-4d6d-a508-5fa818b786ed-sys\") pod \"perf-node-gather-daemonset-lfb9h\" (UID: \"a18b612d-eb27-4d6d-a508-5fa818b786ed\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-lfb9h" Nov 02 23:04:41.854422 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:41.854384 4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57bj7\" (UniqueName: \"kubernetes.io/projected/a18b612d-eb27-4d6d-a508-5fa818b786ed-kube-api-access-57bj7\") pod \"perf-node-gather-daemonset-lfb9h\" (UID: \"a18b612d-eb27-4d6d-a508-5fa818b786ed\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-lfb9h" Nov 02 23:04:41.854654 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:41.854419 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a18b612d-eb27-4d6d-a508-5fa818b786ed-podres\") pod \"perf-node-gather-daemonset-lfb9h\" (UID: \"a18b612d-eb27-4d6d-a508-5fa818b786ed\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-lfb9h" Nov 02 23:04:41.854654 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:41.854442 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a18b612d-eb27-4d6d-a508-5fa818b786ed-lib-modules\") pod \"perf-node-gather-daemonset-lfb9h\" (UID: \"a18b612d-eb27-4d6d-a508-5fa818b786ed\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-lfb9h" Nov 02 23:04:41.854654 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:41.854460 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a18b612d-eb27-4d6d-a508-5fa818b786ed-sys\") pod \"perf-node-gather-daemonset-lfb9h\" (UID: \"a18b612d-eb27-4d6d-a508-5fa818b786ed\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-lfb9h" Nov 02 23:04:41.907678 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:41.907604 4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57bj7\" (UniqueName: \"kubernetes.io/projected/a18b612d-eb27-4d6d-a508-5fa818b786ed-kube-api-access-57bj7\") pod \"perf-node-gather-daemonset-lfb9h\" (UID: \"a18b612d-eb27-4d6d-a508-5fa818b786ed\") " pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-lfb9h" Nov 02 23:04:41.958505 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:41.958464 4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-lfb9h" Nov 02 23:04:42.331751 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:42.331719 4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vlpkd/perf-node-gather-daemonset-lfb9h"] Nov 02 23:04:42.344500 ip-10-0-128-89 kubenswrapper[4935]: W1102 23:04:42.335418 4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda18b612d_eb27_4d6d_a508_5fa818b786ed.slice/crio-7924b7fc7cf67e583f02bf3ebb07e165de48c2b91c52af44e454356bfe9637cc WatchSource:0}: Error finding container 7924b7fc7cf67e583f02bf3ebb07e165de48c2b91c52af44e454356bfe9637cc: Status 404 returned error can't find the container with id 7924b7fc7cf67e583f02bf3ebb07e165de48c2b91c52af44e454356bfe9637cc Nov 02 23:04:42.362485 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:42.362450 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-gh5gd_42f901f6-b299-4cc4-8871-2753cb497f94/manager/0.log" Nov 02 23:04:42.650817 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:42.650722 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-lfb9h" event={"ID":"a18b612d-eb27-4d6d-a508-5fa818b786ed","Type":"ContainerStarted","Data":"c97399ab77bad6816b677eb94a6b7dc489603178beaac949911326f95a45cecb"} Nov 02 23:04:42.650817 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:42.650789 4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-lfb9h" event={"ID":"a18b612d-eb27-4d6d-a508-5fa818b786ed","Type":"ContainerStarted","Data":"7924b7fc7cf67e583f02bf3ebb07e165de48c2b91c52af44e454356bfe9637cc"} Nov 02 23:04:42.650817 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:42.650813 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-lfb9h" Nov 02 23:04:42.755953 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:42.755903 4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-lfb9h" podStartSLOduration=1.755889018 podStartE2EDuration="1.755889018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 23:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:04:42.7554191 +0000 UTC m=+6975.194993824" watchObservedRunningTime="2026-11-02 23:04:42.755889018 +0000 UTC m=+6975.195463737" Nov 02 23:04:42.813106 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:42.813078 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-nmwv9_0d2b5b50-c80c-4bac-8830-6c38da5fee20/proxy/0.log" Nov 02 23:04:43.600275 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:43.600242 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-8xz8j_808ca2fc-b968-4d50-9276-273bedb83a22/manager/0.log" Nov 02 23:04:44.503347 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:44.503318 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-8ggc5_7797919e-d037-4b9c-9eb9-3d0d8af052fb/manager/0.log" Nov 02 23:04:46.224405 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:46.224372 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-dhqhh_c6839ea0-0dae-40a5-a5c3-94220c3c70de/manager/0.log" Nov 02 23:04:46.875900 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:46.875860 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-28jss_cd2913d9-e59b-4b4a-b10a-a324e0c62f25/manager/0.log" Nov 02 23:04:48.325488 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:48.325457 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-nmwv9_0d2b5b50-c80c-4bac-8830-6c38da5fee20/proxy/0.log" Nov 02 23:04:48.671935 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:48.671846 4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vlpkd/perf-node-gather-daemonset-lfb9h" Nov 02 23:04:49.611143 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:49.611093 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 23:04:49.965025 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:49.964949 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-z4kx6_a899c9c5-79b4-472d-942c-5e0c4c53e954/cluster-monitoring-operator/0.log" Nov 02 23:04:51.803874 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:51.803844 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7c689579c-cvz5g_d52ce2cb-6893-4347-b3aa-6268aae6ceda/istio-proxy/0.log" Nov 02 23:04:53.675314 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:53.675286 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-86qhj_2dbc0749-14de-4ad6-bc67-f593966c5a1f/node-exporter/0.log" Nov 02 23:04:54.045052 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:54.045020 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-4r6s4_d9f99d96-f28e-4152-8285-c6aefa58a412/istio-proxy/0.log" Nov 02 23:04:54.048165 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:54.048142 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-4r6s4_d9f99d96-f28e-4152-8285-c6aefa58a412/istio-proxy/1.log" Nov 02 23:04:54.076902 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:54.076880 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-86qhj_2dbc0749-14de-4ad6-bc67-f593966c5a1f/kube-rbac-proxy/0.log" Nov 02 23:04:54.459744 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:54.459668 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-6fb52_b78539b8-fd5e-4fd6-a00d-95857df77139/istio-proxy/0.log" Nov 02 23:04:54.537860 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:54.537833 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-86qhj_2dbc0749-14de-4ad6-bc67-f593966c5a1f/init-textfile/0.log" Nov 02 23:04:55.143915 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:55.143886 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-hpgpb_371ba12b-5ead-4acc-aa5d-1de3d1e97b64/istio-proxy/0.log" Nov 02 23:04:56.198710 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:56.198681 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-74987c9b47-zcnwr_c8acbb0a-9fb9-494f-97d3-05385082fae6/istio-proxy/0.log" Nov 02 23:04:56.373027 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:56.372989 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 23:04:56.835605 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:04:56.835574 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-798b7b6b45-hlcl7_44a209b7-2144-45a0-bea0-2f8adf6658cb/router/0.log" Nov 02 23:04:58.467856 ip-10-0-128-89 kubenswrapper[4935]: E1102 23:04:58.467817 4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2nshl" containerName="extractor" Nov 02 23:05:00.553083 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:05:00.553041 4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mz5qm_3ab0662d-ef78-415d-98f6-8b271e23e16f/konnectivity-agent/0.log" limit=1024 Nov 02 23:05:02.321262 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:05:02.321230 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-gh5gd_42f901f6-b299-4cc4-8871-2753cb497f94/manager/0.log" Nov 02 23:05:02.973344 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:05:02.973306 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-8xz8j_808ca2fc-b968-4d50-9276-273bedb83a22/manager/0.log" Nov 02 23:05:03.583900 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:05:03.583869 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-8ggc5_7797919e-d037-4b9c-9eb9-3d0d8af052fb/manager/0.log" Nov 02 23:05:04.901844 ip-10-0-128-89 kubenswrapper[4935]: I1102 23:05:04.901777 4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7c689579c-cvz5g_d52ce2cb-6893-4347-b3aa-6268aae6ceda/istio-proxy/0.log"